Choosing the Randomisation Unit
By the end of this lesson, you should be able to: choose a randomisation unit and defend it, say what each alternative costs, keep the analysis unit matched to the randomisation unit, and spot the two failures that produce a clean-looking readout from a broken experiment.
Two decisions, not one
Every experiment makes two choices about units, and they're easy to confuse.
What you randomise on decides who gets which variant. What you analyse on decides what counts as one observation.
They must be the same thing. When they aren't, the test breaks in a way that leaves no trace in the output.
Why not the session?
Randomising per session is tempting. Sessions are more numerous than users, so you get more units, which means more power for the same calendar time. You don't need anyone to be logged in.
Here's what it costs, computed on Basket's actual session counts:
| Sessions the user has | Share of users | Chance they see both variants |
|---|---|---|
| 1 | 7.7% | 0.0% |
| 2 | 9.4% | 50.0% |
| 3 | 11.3% | 75.0% |
| 5 | 10.8% | 93.8% |
| 10 | 4.7% | 99.8% |
82.2% of Basket's users would see both variants.
A user who sees the old checkout on Monday and the new one on Wednesday is in neither group. Their behaviour is a blend, and it belongs to no arm. Worse, the inconsistency is itself an experience: a product that changes between visits is confusing in a way that neither variant is.
That's the general rule. Randomise on the coarsest unit the effect could reach, which for most product changes is the person.