Randomisation and What an Experiment Proves
By the end of this lesson, you should be able to: explain what randomisation buys that no amount of analysis can recover, name the two ways a non-experimental comparison goes wrong, put rough numbers on how wrong, and say precisely what an experiment does and does not prove.
Three answers to one question
Basket ships a feature. Its true effect on conversion is +3.0%, and for once we know that, because the number was planted before the data was generated.
Three teams measure it, all competent, all using real data.
| Method | What it says | Off by |
|---|---|---|
| Before launch vs after launch | +19.12% | +16.12% |
| People who used it vs people who didn't | +74.27% | +71.27% |
| A randomised experiment | +3.45% | +0.45% |
The second method is wrong by a factor of twenty-five. Nobody made an arithmetic mistake, and both wrong answers have tight confidence intervals that exclude the truth comfortably.
Why before-and-after fails
The obvious defence of before-and-after is that it's the same product and the same users, so what could differ? Time could.
| Conversion | |
|---|---|
| First week, long before launch | 0.1876 |
| Week 4, still before launch | 0.2127 |
| Whole period before launch | 0.1996 |
| Whole period after launch | 0.2378 |
Conversion rose from 0.1876 to 0.2127 before anybody saw the feature. Something else was pushing it up: seasonality, marketing, a better cohort of signups, word of mouth. Whatever it was, it kept pushing after launch too, and the before-and-after comparison hands the entire trend to the feature.
Look at the left panel. There's no step at the launch line. The feature's real +3% is there, and it's invisible next to a trend several times its size.