Stratified and Blocked Randomisation

By the end of this lesson, you should be able to: say what stratification removes, work out the most it can buy, choose between stratifying and adjusting afterwards, and explain why the answer depends almost entirely on one correlation.

Balanced on average isn't balanced this time

"Random assignment gives balanced groups." That's true on average, over all the assignments you could have drawn. But you run one experiment, with one assignment. In any single draw, one group can end up with more heavy users than the other by chance, and that imbalance shows up as noise in the estimate.

Stratified randomisation removes that chance imbalance by construction. Split users into groups, called strata, using something known before the experiment starts (for example, how much they used the product last month), then randomise within each stratum: half of each stratum to treatment, half to control. The two groups are then forced to have the same mix of light and heavy users.

Blocking is the same idea taken to its limit: pair up the two most similar users, flip a coin to decide which gets treatment, and repeat. Each pair is a stratum of two.

Both keep the estimate unbiased; randomisation still happens within each stratum. What they change is the spread of the estimate from one experiment to the next.

What it can buy: one number decides

The variance stratification can remove depends on a single quantity: $\rho$ , the correlation between the stratifying variable and the outcome. At best, it removes the share $\rho^2$ of the outcome's variance, so the standard error can shrink by at most

$1 - \sqrt{1 - \rho^2}$

Real strata are coarser than perfect knowledge of the covariate, so they capture somewhat less.

Try it: what stratifying on a covariate is worth

Correlation between covariate and outcome (ρ)0.70Number of strata5

Exact for a normally distributed covariate split into equal-sized strata, with half of each stratum randomised to treatment. The ceiling is what perfect knowledge of the covariate could remove.

Standard error cut, 5 strata

25.1%

Ceiling at this correlation

28.6%

1 − √(1 − ρ²)

Users this saves

44.0%

for the same precision

With a correlation of 0.7 and 5 strata, the standard error falls by 25%, close to the 28.6% ceiling, which is the same as getting the precision of 44% more users. Drag the correlation down to 0.1 and the gain is 0.4%: nothing. A covariate that doesn't predict the outcome can't help, however carefully you stratify.

Two things stand out.

The correlation is everything. The difference between a useful and a useless covariate is enormous, and it's set by how well the covariate predicts the outcome, not by how sensible it sounds.

More strata help, with diminishing returns. Two strata capture about two-thirds of what's possible, five capture about 90%, and past ten there's little left to gain. Coarse strata leave some of the covariate's information unused.