Sequential Testing and Peeking
By the end of this lesson, you should be able to: explain why watching a live experiment inflates your error rate, put a number on the inflation for a given number of looks, tell the difference between a fixed-horizon test and an always-valid one, and answer the stakeholder who wants a daily update without either lying to them or blocking them.
The habit nobody thinks is a problem
Basket tested a new delivery slot picker for three weeks. The team had a dashboard, and like every team with a dashboard they opened it every morning.
Here is what they saw, tested fresh each day on everything collected so far.
| Day | Lift | p |
|---|---|---|
| 1 | −6.55% | 0.056 |
| 2 | −2.53% | 0.301 |
| 4 | −0.53% | 0.752 |
| 7 | −0.31% | 0.801 |
| 11 | −1.06% | 0.246 |
| 16 | −0.99% | 0.154 |
| 21 | −0.55% | 0.323 |
Nothing happened. The true effect of that slot picker was exactly zero, and after three weeks of daily checks the experiment correctly said so. Day one came close at p = 0.056, and then it drifted around and settled.
This is the common outcome, and it's precisely why the habit survives. Most of the time you watch an experiment daily and nothing goes wrong, so nothing teaches you to stop.
Now run it a thousand times
One experiment tells you nothing about a procedure. So take the same null experiment, no effect anywhere, and run it a thousand times, checking each one after every day for fourteen days.
| How you read it | Chance of declaring a winner |
|---|---|
| Look once, on the final day | 5.6% |
| Look every day, stop the first time p < 0.05 | 19.0% |
The first row is the test working as designed. You chose , and on null data you get a false positive about five percent of the time. That's the deal.
The second row is the same data read a different way, and it's 3.4 times worse. Roughly one null experiment in five hands you a "significant" result if you're willing to stop the moment you see one.
Nothing about the data changed between those two rows. Only the stopping rule did.