Switchback Experiments
By the end of this lesson, you should be able to: recognise when a user-level split measures the wrong quantity rather than a noisy version of the right one, design a switchback that survives time-of-day, handle carryover, and analyse one at the right unit.
A change you cannot split by user
Meridian smooths surge pricing. The planted effect is a 6.2% reduction in ETA.
Split riders 50/50 and measure. The result:
| Design | Measured | Versus truth |
|---|---|---|
| Switchback, whole city on and off | +7.04% | 114% |
| User-level A/B | +3.47% | 56% |
| Planted truth | +6.20% |
The A/B test found a bit more than half the effect. The natural read is that it's underpowered, so run it bigger:
| Riders | Measured | 95% CI |
|---|---|---|
| 5,000 | +3.83% | [+2.63%, +5.03%] |
| 20,000 | +3.78% | [+3.18%, +4.39%] |
| 80,000 | +3.47% | [+3.17%, +3.78%] |
| 400,000 | +3.64% | [+3.34%, +3.95%] |
The interval tightens around 3.47%, not around 6.20%. More data buys precision on the wrong number.
That's the distinction the whole lesson turns on. The A/B test isn't noisy, it's biased, and no sample size fixes a bias.
Why the control group isn't untreated
Surge smoothing works by changing how drivers are allocated. Smooth it for treated riders and drivers reposition. Those repositioned drivers then pick up control riders too.
So control improves. The measured gap is the difference between two improved groups rather than between treated and untreated, and it comes out at roughly half the truth.
This is interference, and it's structural rather than accidental: any change that acts on a shared resource violates the assumption that one user's assignment doesn't affect another's outcome. Supply allocation, pricing, ranking inventory, matching, and notification volume all qualify.
The fix is to stop trying to have treated and untreated users at the same time. Turn the whole city on and off, alternating over time, and compare the on windows to the off windows. Now there's no untreated rider sharing a driver pool with a treated one, because in any given hour everyone is on the same setting.
