Pre-Registration and the Design Doc
By the end of this lesson, you should be able to: explain why an experiment's analysis has to be decided before launch, list what goes into a design document, describe the checks to run before a test starts, and treat findings you didn't plan for correctly.
The same data, many answers
An experiment produces data. Turning it into a conclusion involves many choices: which metric, which users to include, what to do with outliers, which test to use, which segments to look at, when to stop. Each choice is reasonable on its own. Together, they give a lot of room to reach different answers.
A study published in 2018 measured how much room. Twenty-nine teams of analysts were given the same dataset and the same question: are football referees more likely to give red cards to players with darker skin?
- The teams' estimates ranged from slightly fewer red cards (an odds ratio of 0.89) to nearly three times as many (2.93).
- 20 teams found a statistically significant effect. 9 teams didn't.
- The teams used 21 different combinations of control variables.
- Neither the analysts' expertise nor what they believed beforehand explained the differences.
These were honest analysts making defensible choices, and they still disagreed on whether there was an effect at all. Now imagine one analyst who can see the results while choosing. Every choice can lean, a little, toward the answer they hope for, without any single choice looking wrong.
The fix is simple to state: make the choices before you see the data. Writing them down in advance is called pre-registration.
Why "before" matters so much
A p-value of 0.05 means that, if the change did nothing, a result this extreme would happen about 5% of the time. That's only true for one analysis decided in advance.
If you try several analyses after seeing the data and report the best one, your real false positive rate is much higher. Here's how fast it grows, assuming each analysis gave an independent chance of a false result.
Try five reasonable analyses and pick the best, and you have about a one-in-four chance of finding a "winner" in a change that did nothing.