The OEC and Guardrail Metrics
By the end of this lesson, you should be able to: explain why an experiment needs one primary metric chosen in advance, judge whether a metric is a good primary, choose guardrail metrics and decide how strictly to watch them, and handle the case where the outcome you care about takes too long to measure.
One metric decides
An experiment usually tracks dozens of metrics. If you decide afterwards which one counts, you can almost always find one that moved in the direction you hoped. So before launch, choose:
- One primary metric, sometimes called the overall evaluation criterion (OEC). This decides whether the change succeeded.
- Guardrail metrics. These don't need to improve, but they must not get meaningfully worse.
Everything else is diagnostic: useful for understanding what happened, not for deciding.
The metric that looked perfect
Bing measures its success by query share and revenue. So "queries per user" and "revenue per user" sound like obvious primary metrics.
Then an experiment had a bug that showed users much worse search results. Both metrics went up sharply.
| Metric | Change in the buggy treatment |
|---|---|
| Distinct queries per user | Up more than 10% |
| Revenue per user | Up more than 30% |
The explanation is simple once you see it. With worse results, users had to search again and again to find what they wanted, so queries went up. And with the main results less useful, the ads looked relatively better, so people clicked more ads, and revenue went up. In the short term, the worse product won on both metrics. Over time, users would leave for a competitor.
So Bing's team broke query volume into parts:
- Users is fixed by the experiment's design, because the split decides how many users each group gets. It can't be the primary metric.
- Queries per session is ambiguous. More could mean users are struggling, fewer could mean they gave up. It should be low, as long as people still find what they need.
- Sessions per user is the one to increase. Satisfied users come back more often. Bing made it a key part of its OEC.
Revenue became a constraint rather than the goal: increase it without hurting sessions per user.