Pre-Registration and the Design Doc
By the end of this lesson, you should be able to: write a design doc that makes the readout a formality, state a decision rule that can't be argued with afterwards, explain what pre-registration protects against, and keep exploratory analysis without letting it decide anything.
An experiment where nothing happened
Basket tested a button copy change on 61,184 users. The true effect is exactly zero. Nothing was done to anybody that could have moved any metric.
The pre-registered analysis says:
| Metric | Conversion (chosen before launch) |
| Population | Everyone (chosen before launch) |
| Stop | Day 14 (chosen before launch) |
| Result | p = 0.559, not significant |
Correct. Don't ship. That takes about thirty seconds and there's nothing to discuss.
Now suppose those three choices hadn't been made in advance.
363 ways to analyse one null experiment
An analyst who wants a result has three ordinary-looking freedoms: which metric, which segment, and when to stop. Eleven metrics, eleven segments, three stopping points. Every combination is defensible in isolation.
That's 363 analyses. Five come back with p below 0.05:
| Metric | Segment | Stopped | Lift | p |
|---|---|---|---|---|
| Revenue | Country US | Day 14 | −8.36% | 0.0075 |
| Sessions | Country US | Day 14 | −1.81% | 0.0136 |
| Revenue | Platform android | Day 14 | −8.10% | 0.0143 |
| Revenue | Everyone | Day 14 | −4.65% | 0.0313 |
| Sessions | Everyone | Day 14 | −1.03% | 0.0404 |
"US revenue fell 8.4%, p = 0.0075" is a sentence somebody would put in a deck. It's a large effect, a tight p-value and a segment that matters commercially. It came from an experiment in which literally nothing was done.
Notice something about that table: every one of the five findings is negative. A team that wanted to kill this feature has five ways to justify it, all of them technically correct and all of them meaningless. The freedom to choose your analysis afterwards doesn't only manufacture wins.
Five out of 363 is 1.4%, which is lower than 5% because these analyses overlap heavily. The same users appear in many of them, so they aren't independent tests. The right-hand curve above is the idealised version, and the real inflation is smaller. The conclusion doesn't change: at least one publishable finding exists, and someone looking for one will find it.