Cluster Randomisation and the Design Effect

By the end of this lesson, you should be able to: recognise when you're forced into cluster randomisation, compute a design effect, say why the naive analysis fails so badly, and size a cluster experiment by the thing that actually buys power.

When you can't randomise a person

Meridian tests new driver incentive copy. It can't randomise individual drivers: they talk to each other, they work the same streets, and the incentive changes how busy the whole market is. So it randomises cities. Six treated, six control, 3,000 users each.

The obvious analysis pools all 36,000 users:

The naive standard error is 5.7 times too small.

At the naive SE this is a z of 1.37. At the honest one it's a t of 0.24. Same data, and one of those readouts is a finding while the other is nothing at all.

Look at the left panel. You have twelve numbers, and they overlap. The 36,000 users were never twelve thousand independent pieces of evidence about the incentive copy; they were twelve.

The design effect

Users inside a city are correlated: same weather, same driver supply, same local marketing. The intra-cluster correlation measures how much:

$\text{ICC} = \frac{\text{variance between clusters}}{\text{variance between clusters} + \text{variance within}}$

Meridian's is 0.01. One percent. It sounds ignorable, and here is why it isn't:

$\text{design effect} = 1 + (m - 1)\,\text{ICC}$

where $m$ is users per cluster.

At Meridian's design the effective sample size is 1,162, not 36,000.

And $\sqrt{31.0} = 5.6$, which is exactly the standard error gap measured above. The design effect divides your sample size; its square root multiplies your standard error.

The ICC is small and the cluster is big, so the product is large. That's the whole mechanism, and it's why "the ICC is only 0.01, let's ignore it" is the most expensive sentence in this lesson.