Sizing the Opportunity

By the end of this lesson, you should be able to: put a defensible number on what a problem is costing, measure the counterfactual instead of assuming it, explain why two equally large problems can be worth completely different amounts of effort, and rank work in a way somebody can actually act on.

Finding it is half the job

This track has found two things wrong with Basket. A payment regression in android 8.2, and a channel mix shift that dragged blended conversion down.

Both are real. Neither comes with a price tag attached, and until one does, nobody can decide what to do. "Checkout conversion fell fourteen points in one segment" is a finding. "This is costing us seventy-nine thousand dollars a week" is a decision.

The formula is simple enough to do in your head in an interview:

$\text{Size} = \text{Affected volume} \times \text{Gap to the counterfactual} \times \text{Value per unit}$

All three terms are places to go wrong, and the middle one is where most people go wrong worst.

Measure the counterfactual, never assume it

The gap is the difference between what happened and what would have happened otherwise. That second number isn't available, so it has to be estimated, and the estimate is the whole argument.

The bad version is picking a round number. "Checkout should convert at 90%" invites an immediate and fatal question: says who?

The good version is finding traffic that's genuinely comparable and measuring it. For Basket's payment bug, comparable means everything except the broken cell, over the same days:

Every input is measured. Nobody picked 86.8%; it's what iOS, web and android 8.1 actually did over the same period. That's a number you can defend in a room where somebody disagrees with you, and the round-number version isn't.