Cluster Randomisation and the Design Effect

By the end of this lesson, you should be able to: recognise when you have to randomise groups instead of people, compute a design effect, explain why the naive analysis fails so badly, and size a cluster experiment by the thing that actually buys power.

When you can't randomise a person

An ordinary A/B test assumes that one user's treatment doesn't affect anyone else. Often that's false. Give some drivers a new incentive, and they change how busy the streets are for every driver nearby. Give some LinkedIn members a new feed feature, and their connections see different posts.

LinkedIn's experimentation team described the problem in 2019: when a feature also affects users' connections, it breaks the assumption under nearly every company's experimentation system, and a platform using ordinary random assignment and a two-sample t-test "will not only fail to account for the network effect, but potentially give highly biased results".1 Facebook described a system "deployed at Facebook, that accounts for interference between experimental units through cluster-randomized experiments", and reported case studies with significant network effects.2

The standard fix is cluster randomisation: randomise whole groups (cities, stores, schools, clusters of connected friends) so that the people who affect each other get the same treatment. The interference lesson covers why. This lesson covers the price.

The design effect

Users in the same cluster are alike: same city, same weather, same local supply of drivers. The intra-cluster correlation (ICC) measures how alike:

$\text{ICC} = \frac{\text{variance between clusters}}{\text{variance between clusters} + \text{variance within clusters}}$

An ICC of 0.01 means only 1% of the variation is between clusters. That sounds ignorable. It isn't, because of the design effect:

$\text{design effect} = 1 + (m - 1)\,\text{ICC}$

where $m$ is the number of users per cluster. With 3,000 users per city and an ICC of 0.01, the design effect is 31. Twelve cities with 3,000 users each, 36,000 users in total, carry about the same information as 1,160 independent users.

The design effect divides your sample size; its square root multiplies your standard error. $\sqrt{31} = 5.6$, so a standard error computed as if the 36,000 users were independent is more than five times too small.

A small correlation times a big cluster makes a big number. That's the whole mechanism.