The OEC and Guardrail Metrics
By the end of this lesson, you should be able to: pick a primary metric on evidence rather than instinct, work out whether a candidate can resolve the effect you expect, recognise the metrics that look ideal and are disqualified, and set guardrails that constrain a decision instead of decorating it.
One number decides
Every experiment needs one metric that decides it. Not a scorecard, not a dashboard. One number, named before launch, that the ship decision is a function of.
The reason is lesson 15's arithmetic: if three metrics can each justify shipping, you've tripled your false positive rate and you've also given yourself permission to pick whichever one came out well. A single primary is what makes a result a decision rather than a negotiation.
The usual name for it is the OEC, the Overall Evaluation Criterion. What matters is the discipline, not the acronym.
The test most candidate metrics fail
"Pick a metric that reflects long-term value" is true and useless. Here's a criterion you can actually apply: can this metric resolve the effect you expect, in the time you have?
Basket has about 42,000 users a week available for one experiment. For each candidate, the smallest effect detectable at 80% power:
| Candidate metric | Spread (CV) | 1 week | 2 weeks | 4 weeks |
|---|---|---|---|---|
| Sessions per user | 0.63 | 1.71% | 1.21% | 0.85% |
| Conversion rate | 2.08 | 5.69% | 4.03% | 2.85% |
| Revenue per user, capped p99 | 2.53 | 6.92% | 4.89% | 3.46% |
| Revenue per user | 2.72 | 7.43% | 5.25% | 3.71% |
Now suppose you expect a 2% effect, which is a perfectly normal size for a real product change:
| Candidate metric | Weeks needed | Verdict |
|---|---|---|
| Sessions per user | 0.7 | usable |
| Conversion rate | 8.1 | slow |
| Revenue per user, capped p99 | 12.0 | slow |
| Revenue per user | 13.8 | not viable |
Revenue per user is the metric the business cares about most, and it needs three and a half months to answer a question about a 2% change. That's not a metric you can run a product on.
This is the trade at the heart of metric choice. The numbers closest to money are the noisiest, and the numbers you can measure quickly are further from money. Anyone who tells you to just use revenue hasn't done this arithmetic.