Ratio Metrics and the Delta Method

By the end of this lesson, you should be able to: say why the obvious standard error for a ratio is wrong, write down and use the delta method, check any standard error against a bootstrap, and judge from your own data how much the shortcut would cost you.

A metric with two moving parts

Basket's search team measures click-through rate: clicks divided by impressions. Over the experiment that's 97,658 clicks on 1,156,407 impressions, so the CTR is 0.08325.

Now ask what the standard error of that number is. The obvious answer treats it as a proportion, because it looks exactly like one:

$\text{SE}_{\text{naive}} = \sqrt{\frac{r(1-r)}{\sum_i D_i}} = \sqrt{\frac{0.08325 \times 0.91675}{1{,}156{,}407}} = 0.000363$

That formula assumes 1,156,407 independent Bernoulli trials. Both halves of that assumption are false.

Impressions are not independent. They come in clumps of one user, and a user who clicks a lot clicks a lot across all of their impressions.

The denominator is not a fixed constant. It's a random quantity that varies enormously between users.

Basket's users get anywhere from 1 to 68 impressions, averaging 15 with a standard deviation of 9.3. Randomisation gave each arm a random draw of users, so it also gave each arm a random total number of impressions. The naive formula never accounts for that.

Checking it against something that can't be wrong

Rather than argue about which formula is right, use a method that makes no assumptions at all. The bootstrap: resample users with replacement, recompute the pooled ratio, and look at how much it moves.

That's the definition of a standard error, computed directly. It's slow and it needs no theory, which makes it the perfect referee.

The delta method lands on the bootstrap to three decimal places. The naive formula is 7% too small, which means every confidence interval built on it is 7% too narrow and every p-value slightly too confident.