Long-Term Holdouts and Effect Decay
By the end of this lesson, you should be able to: tell decay from a permanent gain, say when a holdout can first rule out pure novelty, size a holdout against what it costs, and name the reason a long-running holdout stops being valid.
The same experiment, read at different times
Meridian launches a loyalty programme to 80% of riders and holds 20% back. Trips per user, treated against holdout:
| Read at | Measured lift | Versus the 26-week answer |
|---|---|---|
| Week 1 | +10.2% | 4.2x |
| Weeks 1-2 | +9.1% | 3.8x |
| Weeks 1-4 | +7.5% | 3.1x |
| Weeks 5-8 | +4.3% | 1.8x |
| Weeks 13-16 | +2.8% | 1.2x |
| Weeks 23-26 | +2.5% | 1.0x |
Every one of those numbers is a correct measurement of a real randomised comparison. They differ because the effect itself changed, not because any of them was computed wrongly.
A team that ran two weeks, saw +9.1%, and put that in the annual plan has booked roughly four times the value the feature delivers. The two-week readout contains no signal that this will happen.
Decay or dilution
Here's the distinction that decides what you do, and the reason a holdout has to keep running.
A novelty effect decays to nothing. Users tried the loyalty programme because it was new, the interest wore off, and the long-run value is zero. Ship it and you've added maintenance for nothing.
A real effect that settles decays to a smaller, permanent level. The initial spike was novelty on top of a genuine gain. Ship it and you get 2.4% forever.
Those two stories look nearly identical early:
| Week | Measured | If it decays to zero | If it settles at 2.4% |
|---|---|---|---|
| 1 | +10.2% | +9.0% | +9.0% |
| 2 | +8.1% | +7.2% | +7.7% |
| 3 | +6.8% | +5.8% | +6.6% |
| 6 | +4.7% | +3.0% | +4.6% |
| 10 | +2.9% | +1.2% | +3.3% |
| 18 | +3.2% | +0.2% | +2.6% |
| 26 | +2.7% | +0.0% | +2.4% |
At week 3 the two stories differ by 0.8 points and no measurement can separate them. By week 18 they differ by the entire remaining effect.
That gap is what the extra fifteen weeks of holdout bought. It is not precision on a number you already had; it is the difference between "ship it" and "delete it".