Choosing the Randomisation Unit
By the end of this lesson, you should be able to: name the common randomisation units and when each fits, explain how users get assigned to groups, choose a unit for a given product change, and explain why the analysis has to happen at the same level as the randomisation.
Two decisions, not one
Every experiment makes two choices about units.
- The randomisation unit: the thing that gets flipped into treatment or control. A user, a device, a session, a team, a city.
- The analysis unit: what one row of your analysis represents. A user, a session, a page view, an order.
People usually think hard about the first and forget the second. Both matter, and they need to line up.
The common randomisation units
| Unit | When it fits | The catch |
|---|---|---|
| Logged-in user | Most product changes, when users sign in | Only covers signed-in traffic |
| Cookie or device | Logged-out products, short tests | One person on several devices can land in both groups |
| Session or page view | Changes users won't notice switching, like back-end speed | The same person sees both versions |
| Cluster (team, company, school) | Collaboration features, enterprise products | Far fewer units, so less power |
| Region or time window | Marketplaces where users share supply, like drivers | Very few units, needs special designs |
Microsoft's experimentation team points out three situations that push you away from the plain user.
- IDs that don't last. Cookies get cleared and replaced, so the same person can turn into a "new" user halfway through. That makes cookies fine for short tests but poor for measuring long-term effects. It also means you should check whether one version makes people lose their cookies more often than the other.
- Users who affect each other. If treated users change what control users experience, through messages, shared content or shared supply, the comparison stops being clean. A common fix is to randomise whole clusters of connected users together. Lesson 17 covers this.
- Products that must look the same across a company. Some business software has to give everyone in an organisation the same experience. Then you have to randomise whole organisations, which means far fewer units.