Define the Metric Before You Move It
By the end of this lesson, you should be able to: explain why two honest analysts report different numbers for the same metric, name the three questions every definition has to answer, and work backwards from a decision to a metric instead of forwards from whatever the logs happen to contain.
Four teams, four numbers, and all of them are right
Meet Basket, the product we will use for this whole track. Basket is a grocery delivery app running in five countries. People open the app, search, add things to a cart, start checkout, and place an order. Every number in every lesson here is computed from Basket's event table, so when something adds up, it really adds up.
Leadership asks what sounds like the simplest question in analytics. How many daily active users does Basket have?
Four teams go away and answer it. Nobody lies. Nobody makes a mistake. All four run their query against the same table.
| Who ran it | What they counted | Answer |
|---|---|---|
| Mobile team | Devices that opened the app, UTC day | 4,746 |
| Data platform | People who opened the app, UTC day | 4,455 |
| Growth | People who opened the app, in the user's own day | 4,690 |
| Product | People who did something past opening, in their own day | 3,119 |
The biggest answer is 52.2% above the smallest. That isn't a rounding difference. That's one team reporting nearly half again as many users as another team, on the same Tuesday.
The gap is a decision, not a rounding error
It's tempting to treat this as a pedantic argument for the data team to sort out quietly. It isn't. Look at what each number does.
Say the annual goal is 6,000 daily active users. Under Growth's definition Basket needs to grow 28% and the team is fine. Under Product's definition it needs to grow 92% and somebody is going to lose headcount. Nothing about the product changed. Only the WHERE clause did.
Now imagine a feature ships and daily actives go up 4%. Did it work? If the feature makes people open the app more often but does nothing once they are inside, then it moves three of these four numbers and leaves the fourth flat. Which team you ask decides whether the feature launches.
This is the real reason interviewers open with "how would you measure that." They aren't testing whether you can name a metric. They are testing whether you know that naming one is a choice with consequences, and whether you'll make that choice out loud instead of assuming everyone means the same thing.
In an interview, say the definition before you say the number. "I will count a user as active if they took an action past opening the app, on their own calendar day" costs you eight seconds and separates you from most candidates immediately.