Manzi: Guest Columns: Evidence-based Management (EBM)

Evidence-Based Management… in Retail?

The retail channel is full of ideas to improve profits, and many of them are about spending money to make money. From building new stores to remodeling existing ones, from buying TV advertising to running in-store promotions, from introducing new products to training new staff, any new program that works across a large network of stores can generate a lot of cash. Unfortunately, the converse is also true - a new program that does not live up to expectations can have a devastating impact on profits if it is rolled out across the network and fails.

The central issue in applying the principles of evidence-based management in these situations is the correct identification of causality. The causal impact of most programs is small compared to natural variation in performance, and the total number of data points is very small. Most correlations determined through historical analysis are spurious, and it is impossible to know when this is not the case.

In 1999, I founded APT to help address this problem. APT helps large consumer-focused organizations apply an evidence-based management process to identify, quantify and exploit causal relationships between management actions and financial outcomes, known as Test & Learn™. It is now being used by a growing number of retailers, consumer goods manufactures and financial services institutions to increase the effectiveness, profitability and speed to market of new programs and capital investments. When I say “Test and Learn,” I am referring to the process of testing an idea in a small number of stores to understand its impact before rollout.

The basic steps of the Test & Learn™ process are not complicated – in fact they’re intuitively obvious:

Establish a Hypothesis
Design a Test
Execute the Test
Analyze the Test
Plan the Rollout
Execute the Rollout

However, when a test is executed in an ad hoc manner, a serious challenge manifests itself at a critical stage of the process. That stage is when it comes time to spend money to rollout the idea. If the executive in charge of that decision is not sufficiently confident that the idea is going to work, he or she simply won’t approve the rollout.

Even more frustrating to management is that there is no way to accurately predict what customers will do in response to a new program before it’s launched. One can execute deep customer research, draw on analogous experience from other retailers, create virtual reality simulations, or consult one’s gut, but no research-based approach can reliably answer the question “what will my returns be if I make this investment?”

The only way to know for sure is to treat the initial implementation of the rollout as a test. A set of stores should be selected that is large and diverse enough to present an accurate cross-section of the network. The performance of these stores should be carefully tracked and compared to the performance of a well-matched control group, to capture the attributable impact of the program’s components, and truly measure the investment returns.

Both the size of the stakes and the complexity of gleaning actionable insights from in-store experimentation grow geometrically as the network grows. Often times these organizations are running multiple tests of important retail ideas at any given time. The ideas tested cross the enterprise, from marketing and merchandising, through operations and HR, to investments in capital improvements.

Up until the last ten years, in-store experimentation was practiced by special occasion only (if at all) by nearly all large network retailers, due to the nearly insurmountable challenges of mobilizing, tracking and analyzing the massive amounts of data required. Over the last ten years, however, the explosion in computing power (and the steep decline in its cost) has profoundly changed the face of retail analytics, allowing large retailers to make the experimentation process both more rigorous and systematic. Today, a best-practice Test & Learn process reliably, consistently and economically answers three questions about any tested store program before rollout:

What impact will this idea have on my results – revenues, profits, investment returns, market share, etc.?
Will the idea have a larger impact on some stores than others? What is the predicted impact by store and by market, and can I design a rollout program that maximizes returns?
What value is created by the individual components of the idea? Can the idea be engineered before rollout to maximize its value?

By institutionalizing a strong organizational process, Test & Learn combines the evidence based management principles of being committed to “fact based” decision making, encouraging experimentation, and utilizing proper testing techniques while decreasing the potential for weak recommendations.

I’ve watched as best-practice Test & Learn practitioners have showcased the value of this style of decision making, generating profit improvements in the tens of millions of dollars per year, rapidly fine-tuning their responses to market developments, and achieving competitive advantage that is difficult for their retail adversaries to see and even more challenging for them to copy. And they’re just getting started…

Posted on October 1, 2008