CUPED and Variance Reduction

By the end of this lesson, you should be able to: explain why a pre-experiment covariate can shrink your confidence interval without biasing anything, compute the adjustment yourself, predict the gain from a correlation before running anything, and say honestly when it isn't worth the trouble.

The problem, in one number

Basket tested a free delivery threshold on 68,286 users. The metric is revenue per user, and revenue is the worst-behaved number on the platform: four fifths of users spend nothing at all, and the top 1% hold 17.4% of the total.

That noise has a price. It's paid in sample size, and it's paid every single time you test anything with money in it.

Now here's the thing that should feel slightly unfair. Basket's platform already stores, for every one of those users, how much they spent in the four weeks before the experiment started. Nobody had to build anything. The column is just sitting there.

That column knows something.

Users in the bottom tenth of pre-period spend went on to spend about $3.50 during the test. Users in the top tenth spent about $30. The correlation between the two columns is 0.324.

CUPED is the trick that turns that correlation into sample size.

The idea before the formula

Every user's revenue during the test has two parts: the part you could have guessed from what they did before, and the part you couldn't. Only the second part is news. The first part is just that user being who they always were, and it's identical in both arms because randomisation made the arms identical.

So subtract it. Take out the predictable component, keep the surprise, and test that instead.

Nothing about the treatment effect lives in the predictable part, because the pre-period happened before the treatment existed. You can remove it without removing any of the signal, and what leaves with it is pure noise.