Novelty and Primacy Effects
By the end of this lesson, you should be able to: recognise a novelty effect from the shape of a daily readout, explain why running the test longer does not fix it, compute the number that would survive a launch, and tell novelty apart from a real effect that happens to be fading.
A banner that worked, briefly
Basket put a promotional banner on the home screen and measured whether users placed an order that day.
| Day | Control | Treatment | Lift | p |
|---|---|---|---|---|
| 1 | 0.0562 | 0.0636 | +13.15% | 0.0002 |
| 2 | 0.0577 | 0.0597 | +3.59% | 0.299 |
| 3 | 0.0588 | 0.0627 | +6.77% | 0.049 |
| 7 | 0.0567 | 0.0605 | +6.75% | 0.054 |
| 14 | 0.0585 | 0.0594 | +1.51% | 0.658 |
| 21 | 0.0595 | 0.0613 | +2.96% | 0.382 |
Day one is a thumping win with a p-value of 0.0002. By week three the daily numbers are bouncing around a value that isn't distinguishable from zero.
Group them into weeks and the pattern is cleaner than the daily noise suggests:
| Week | Lift | 95% interval | p |
|---|---|---|---|
| 1 | +3.86% | [+1.31%, +6.41%] | 0.003 |
| 2 | +0.56% | [−1.99%, +3.10%] | 0.668 |
| 3 | +0.55% | [−1.98%, +3.08%] | 0.670 |
That's a novelty effect. Users noticed something new, poked at it, and then went back to behaving exactly as they had before.
Running longer does not fix it
The instinct is that a two-week test would've caught this. It wouldn't have. Here is the same experiment read at four different moments, using everything collected up to that point.
| Read after | Lift you would report | 95% interval | Verdict |
|---|---|---|---|
| 3 days | +7.78% | [+3.85%, +11.72%] | Ship |
| 7 days | +3.86% | [+1.31%, +6.41%] | Ship |
| 14 days | +2.21% | [+0.41%, +4.01%] | Ship |
| 21 days | +1.66% | [+0.19%, +3.12%] | Ship |
Every stopping point ships it. Even the full three weeks.
The reason's worth sitting with, because it's the thing most people get wrong about novelty. The cumulative average includes the novelty period forever. Week one's +3.86% is in the numerator no matter how long you run, and adding flat weeks after it dilutes that spike without ever removing it. The reported number drifts down toward zero and never arrives.
So "run it for a month" isn't a defence. A month of data would have reported roughly +1.2% with an interval still excluding zero, and shipped a feature whose steady-state value is nothing.