Building an Experimentation Platform

By the end of this lesson, you should be able to: explain why careful individual experiments don't add up to trustworthy results across a company, set an error budget and say what it costs, order a platform's checks by the damage they prevent, and describe the readout a platform should produce.

What experimentation looks like at scale

At large companies, experimentation is infrastructure, not a project. Booking.com reported in 2017 that its staff "run and analyse more than a thousand concurrent experiments" every day, and that "every change, from entire redesigns and infrastructure changes to bug fixes, is wrapped in an experiment." Microsoft's Bing ran more than 200 experiments at the same time, each typically exposing several million users.

At that scale, the problems change. The question is no longer "is this one test analysed correctly?" but "what does the whole system produce, when thousands of correct tests run every year?"

The arithmetic nobody runs

Suppose every experiment is designed, powered and analysed correctly, at a 5% significance level, and each reads 20 metrics. Even when a change does nothing at all, each metric has a 5% chance of coming out "significant". Across 20 metrics:

$1 - 0.95^{20} = 64\%$

Most experiments that change nothing will still show at least one significant metric. Nobody peeked, picked the metric after the fact, or made a mistake. It's arithmetic. The Bing paper names this directly: at its scale, the team faces "a high occurrence of false positives that we address".

Try it: what a year of correct experiments produces by chance

Metrics read per experiment20Experiments a year220

Significance level per metric:

For an experiment where nothing changed, at the chosen significance level per metric, treating metrics as independent.

False significant readings

220 a year

from changes that did nothing

Experiments with a false winner

64%

Power on a typical real effect

59%

59% at α = 0.05

Real effects missed

41%

the price of a stricter α

With 220 experiments a year and 20 metrics each, there are about 220 false significant readings a year from changes that did nothing. Individual rigour is necessary, and it doesn't add up to trustworthy results across the company. The platform's job is everything between the readout and the launch.