CUPED and Variance Reduction

By the end of this lesson, you should be able to: explain the idea behind CUPED, apply its formula, say why it doesn't bias the result, choose a good covariate, and recognise when it won't help.

The idea

Why is an experiment noisy? Mostly because users are different from each other. Some search 50 times a week, some search twice. When users are split at random, one group happens to get slightly more heavy users than the other, and that difference shows up as noise in the result.

But you already know which users are heavy. You saw how they behaved in the weeks before the experiment started. A user who searched 50 times last week will probably search a lot this week too, with or without the change.

CUPED (Controlled-experiment Using Pre-Experiment Data) uses that knowledge. It removes the part of each user's metric that was predictable from their past behaviour, and compares the groups on what's left. What's left is much less noisy, so the same experiment can detect smaller effects.

It was developed by Microsoft's experimentation team and published in 2013, using Bing's experiments.1

The formula

For each user, let $Y$ be their metric during the experiment and $X$ the same metric from before it started. The adjusted metric is:

where $\bar{X}$ is the average of $X$ over all users, and

is the same as the slope from regressing $Y$ on $X$. Then you compare treatment and control on $Y_{\text{adj}}$ instead of $Y$, using the usual test.

The variance of the adjusted metric is the original variance multiplied by $1 - \rho^2$, where $\rho$ is the correlation between $X$ and $Y$. A correlation of 0.7 removes about half the variance. Since the users needed are proportional to the variance, that halves the users you need.