Switchback Experiments

By the end of this lesson, you should be able to: recognise when a user-level split measures the wrong thing rather than a noisy version of the right thing, design a switchback that survives the time of day, handle carryover, and analyse one at the right unit.

A change you can't split by user

Some changes act on a resource that every user shares: the pool of available drivers, the delivery couriers, the stock of rooms. Give half the users a change and it alters the shared resource for the other half too.

Nicholas Chamandy of Lyft worked through a small example in 2016.1 Two riders open the app at once, with one driver nearby, and Lyft is considering paying the "Prime Time" surge charge on riders' behalf. In his simple model, subsidising Prime Time for everyone raises completed rides by a third. But a standard A/B test, giving the subsidy to one rider and not the other, measures a much bigger gain, because the subsidised rider grabs the one driver that the unsubsidised rider would otherwise have used. The treated group wins partly by taking rides from the control group. Lyft's estimate came out too large "by a factor of 6".

That's interference: one user's treatment changes another user's outcome. It makes a user-level A/B biased, not just noisy, and no sample size fixes a bias. Collecting more users only narrows the interval around the wrong number. The direction depends on the change: Lyft's example overstates the effect, while a change that helps everyone through better driver positioning makes control look better too and understates it. Interference covers the general problem.

Switch the whole system, over time

The fix is to stop having treated and untreated users at the same moment. Turn the change on for everyone in an area for a while, then off, then on, and compare the "on" periods with the "off" periods. In any given period, nobody untreated is sharing a driver with someone treated.

DoorDash's dispatch team adopted this in 2018 and called it switchback testing.2 Testing "SOS pricing", which raises delivery fees when there are too few couriers, they switched it on and off every 30 minutes. Two details made it a proper experiment:

• Each window is randomised separately, by its own coin flip, rather than alternating on a fixed schedule. Each time window is then a randomised unit.
• Regions are randomised independently, so one city can be on while another is off. The units become "time-region units".

They settled on 30-minute windows, using regions "roughly at the city-level". The post also notes that the method was first used in agriculture, for dairy cow experiments.