Sequential Testing and Peeking
By the end of this lesson, you should be able to: explain why checking a test repeatedly and stopping at the first significant result inflates false positives, name the methods that let you look early safely, and tell the difference between stopping early for harm and stopping early for a win.
The habit
Most teams check their experiments every day. That's natural: people want to know whether their idea is working. The problem starts when the check becomes a decision: "it's significant, let's stop and ship."
This is called peeking. It feels harmless, because each individual check uses a correct p-value. It isn't harmless.
Why it inflates false positives
A p-value of 0.05 means a 5% chance of a false positive for one check at a sample size decided in advance. Every additional check is another chance for random noise to cross the line. And if you stop the first time it crosses, you keep every lucky crossing and never see it drift back.
Try it on simulated A/A tests, where nothing differs between the groups.
Checking once at the end gives about 5% false positives, as it should. Checking every day for four weeks and stopping at the first p below 0.05 gives about 27%. More than one A/A test in four would be declared a winner.
Researchers working with Optimizely made the same point about real A/B testing practice: when users continuously monitor their tests, even with 10,000 samples, the false positive rate can easily increase fivefold.