Surrogate Metrics & the OEC

By the end of this article, you should be able to: separate the decision metric from the early signal that stands in for it, explain the condition a surrogate has to meet, check one against past experiments, and recognise the surrogate paradox before it ships something harmful.

The outcome arrives too late

The outcomes that matter most take the longest to see. Whether a member is still subscribed in two months, whether a customer comes back next year, whether a job programme raised earnings over a decade. An experiment that waits for them is slow, and a company that runs slow experiments runs few of them.

Netflix put a number on that cost. Its personalisation tests measured the outcome over 63 days. Deciding after 14 days instead would, at the extreme, increase total experimentation capacity by 300%: four tests in the time of one.1

So two questions have to be answered before any test starts:

1. Which number decides? One metric, agreed in advance.
2. What can you measure early that tells you something true about that number?

The first is the OEC, the overall evaluation criterion: the single metric (or weighted combination) the team agrees will decide the test, chosen before any results exist. It has its own lesson, the OEC and guardrails, because choosing it after the results come in turns twenty metrics into twenty chances to declare a win.

The second is a surrogate, and it's what the rest of this article is about.

What a surrogate is

A surrogate is an early measurement you use in place of the outcome you care about. Medicine has used them for decades, such as tumour size in place of survival in cancer trials. Product teams use them every time they read "day-7 activity" as a stand-in for retention.

A surrogate is only valid under one condition, called the surrogacy assumption: once you know the surrogate, the treatment tells you nothing more about the long-term outcome.2 Ross Prentice's 1989 definition puts it as the surrogate having to "capture" any relationship between the treatment and the true outcome. In plain words, the change has to affect the outcome only through the surrogate. If a change moves day-7 activity and that's the whole story, day-7 activity is a fair stand-in. If it also affects retention some other way, the surrogate misses that part.

That condition is strong, and a single early metric rarely meets it. Susan Athey, Raj Chetty, Guido Imbens and Hyunseung Kang proposed combining many early measures into one surrogate index.3 The steps:

1. Fit a model on historical data that predicts the long-term outcome $Y$ from the early measures $S$: $\hat\mu(S) = \hat{\mathbb{E}}[\,Y \mid S\,]$. Here $\hat\mu(S)$ is the predicted long-term outcome for a user with early measures $S$.
2. Apply it to every user in the experiment, giving each one a predicted long-term outcome.
3. Compare the arms on that prediction:

$\widehat{\text{effect}} = \overline{\hat\mu(S)}_{\text{treatment}} - \overline{\hat\mu(S)}_{\text{control}}$

If the early measures together satisfy the surrogacy assumption, this equals the effect on the long-term outcome. Several measures are much more likely to capture all the paths than any single one.

Their test case was a job-training experiment in California, where the outcome of interest took nine years to observe. Using employment in the first six quarters as surrogates, they could have estimated the programme's long-term effect on employment years earlier, with standard errors 35% smaller than the direct estimate.