Sizing the Opportunity
By the end of this lesson, you should be able to: put a defensible number on what a problem costs, measure the gap instead of assuming it, explain why the size of a problem is different from what you can win back, and rank several opportunities on the same basis.
The sizing formula
"Checkout conversion fell 14 points on one Android version" is a finding. "It's costing us $80,000 a week" is something a team can make a decision on. Sizing turns the first into the second.
Almost every sizing has the same three parts.
- Affected volume. How many users, sessions or orders the problem touches. Only count the ones actually affected, not the whole product.
- Gap. The difference between what happened and what would have happened without the problem. This is the hard part.
- Value per unit. What each lost unit is worth, like the average order value, or the revenue from one more search.
Then turn it into a run rate: the cost per week or per month. "$300,000 so far" starts an argument about the time window. "$80,000 a week until it's fixed" tells everyone what a week of delay costs.
Measure the gap, don't assume it
The gap compares what happened with what would have happened otherwise. That second number is never directly observed, so it has to be estimated. The estimate is the whole argument.
The weak way is to pick a number. "Checkout should convert at 90%." The first question anyone will ask is "says who?", and there's no good answer.
The strong way is to find a comparable group and measure it. Usually that's the users who weren't affected, over the same days: other app versions, other platforms, or a control group in an experiment.
Google did this in 2009 to find out what slow pages cost. Instead of guessing, it ran an experiment: some users got search results slowed down by 100 to 400 milliseconds, and the rest didn't. Comparing the two groups measured the gap directly.
- Slowing results by 100 to 400 milliseconds cut searches per user by 0.2% to 0.6%.
- The loss grew over time. With a 400-millisecond delay, searches fell 0.44% in the first three weeks and 0.76% in weeks four to six.
- It lasted after the fix. Users who'd had the longer delay were still searching 0.21% less five weeks after the delay was removed.
0.6% sounds small, but multiply it by Google's volume and it's a very large number. And the experiment shows two things a guessed number would have missed: the cost of a problem can grow the longer it runs, and some of it doesn't come back straight away once you fix it. So a sizing based only on the first week will usually understate the cost of leaving a problem in place.