The Concept of EVI: Expected Value of Information

If you have a lot of uncertainty now, you don’t need much data to reduce uncertainty significantly…if you know almost nothing, almost anything will tell you something.

 -- Douglas Hubbard in How to Measure Anything (p. 162)

There is a tremendously useful concept, Expected Value of Information (EVI), that I’d like to summarize. 

Assume that you’re faced with a set of choices. In product development, we typically have a long backlog of ideas, and we need to choose a few projects to implement in our next development cycle. The prioritization may be done by the program manager or the group manager, or by some crowdsourcing of votes by several trusted experts.

Given the poor success rate of ideas that we implement, where only about 10-35% of controlled experiments improve the metrics they were designed to improve (Online Controlled Experiments at Large Scale, Tenet 3), the question is what can we do to reduce the development costs?

To be clear up front, I’m not going to claim I have a magic solution for improving that 10-35% rate---I don’t. Significant changes to that percentage are unlikely. What I am going to show is something simple: have smaller initial discovery projects that cost less, and they’ll enable you quickly hone on the more promising projects. If you fail quickly on smaller projects, you’ll have more resources to invest in the ones showing success.

Let’s assume that we know what we’re optimizing for, and we have a clear Overall Evaluation Criterion (OEC). If there is no agreed-upon OEC, then we should step back and define one (or a few key metrics); otherwise, we can declare success now. Note that the OEC can be quantitative data (e.g., user actions like purchases and time to those actions), survey data (are users satisfied), and expert opinion (e.g., a “coherency” score for the design of the user flow).

Now further assume that we have a mean and a 95% confidence interval for the OEC of each project, say by crowdsourcing votes of our trusted experts and merging them.  Every project has uncertainty about its outcome: some projects may have a narrow confidence interval because we know they previously worked (e.g., most ideas that worked in the US have a high probability of working in the UK, so our confidence interval may be narrow), others have more uncertainty represented by a wide confidence interval where the delta OEC (treatment effect) crosses zero (so we do not know if it’s going to be positive or negative for the OEC).

The usual (simplified) process is to prioritize the projects based on the mean of the confidence interval and choose to implement those at the top, based on our available resources.

This is where the concept of EVI comes in. Could we update the mean and confidence interval by getting additional information? What would be the expected cost of that information (ECI)?

Here are some examples of ways to reduce the uncertainty:

1. Expert opinion. Ask some experts in the domain. If they know more than you do, there’s a good chance you can reduce the uncertainty. 
2. Get the relevant data. Many times, a few queries can help assess the coverage of the feature (how many users would see it) and provide upper bounds on the utility. Other types of data may be prior experiments that were run. The name “Discovery project” is sometimes used to identify this step.
3. Conduct user studies and market research. Review sketches, mockups, and prototypes of the idea. Conduct survey and usability studies.
4. Run a simple/cheap controlled experiment. Some of the biggest breakthroughs resulted from a trivial experiment that took days to implement, including for example:

• Long ad titles at Bing, an experiment that increased revenue by over $100M took days to implement (see the opening example in the Surprising Power of Online Experiments and in Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing).
• Optimizing colors, which was worth about $200M for Google (Quora).
• Amazon’s credit card offer, worth tens of millions of dollars (Seven Rules of Thumb).
• Opening links in new tabs, which dramatically improved click-throughs (Seven Rules of Thumb). 

The concept of Minimum Viable Product (MVP) championed by Eric Ries is clearly aligned with the idea of lowering the initial cost to understand the value of the feature.

Doug Hubbard’s wonderful book, How to Measure Anything, covers this idea of quantifying the value of information in Chapter 7. Here is the key diagram:

The x-axis ranges from no information to perfect certainty. The y-axis ranges from zero value to perfect information for the EVI curve, and from zero cost to high cost for the ECI curve.

Next time you are faced with the need to estimate the value of a feature or project, or a choice, think about whether you can invest in a quick discovery project that has small cost—low on the ECI curve-and has high value in reducing uncertainty—high on the EVI curve.

-- Ronny Kohavi