Ratio Metrics and the Delta Method
By the end of this lesson, you should be able to: recognise a ratio metric, explain why its obvious standard error is too small, calculate the correct one with the delta method, and report a ratio metric so its movement can be understood.
What a ratio metric is
Many product metrics divide one total by another.
- Click-through rate per page view: total clicks ÷ total page views.
- Average order value: total revenue ÷ total orders.
- Queries per session: total queries ÷ total sessions.
What makes these tricky is that the denominator isn't the thing you randomised. You randomised users, but the metric is per page view, per order or per session. Each user contributes a different number of page views, and both the top and bottom of the ratio can change with the treatment.
This is common. Microsoft's experimentation team notes that most user-randomised experiments on its platform include page-level metrics alongside user-level ones.
Why the obvious standard error is wrong
The obvious way to get a standard error for click-through rate is to treat every page view as a separate yes/no trial. With 200,000 page views and a 20% click rate, the standard error would be , about 0.09 percentage points.
But page views aren't independent trials. Some users click a lot, and some hardly click at all, and each user brings that habit to every page they view. So 200,000 page views from 20,000 users carry much less independent information than 200,000 separate people would. The real standard error is bigger, and treating page views as independent makes results look more certain than they are.
How much does that matter? It depends on how much users differ. Here are the results of 3,000 simulated A/A tests for each scenario, with 2,000 users per group and about nine page views each. Nothing differs between the groups, so a correct method should come up significant 5% of the time.
When every user has the same click rate, the two methods agree. As users become more different, which is how real users are, the simple method's false positive rate climbs to 18% and then 36%. The delta method stays at 5%.