Stratified and Blocked Randomisation
By the end of this lesson, you should be able to: say what stratification actually removes, compute the ceiling on what it can buy, choose between stratifying and adjusting afterwards, and explain why a coarse stratum captures less than the covariate is worth.
The defence of simple randomisation, and what it leaves out
"Random assignment gives balanced groups." True, and incomplete in a way that matters.
Take Meridian's 60,000 riders and their pre-period trip counts. Assign treatment by coin flip, 4,000 separate times, and measure the gap in prior trips between the two groups each time:
| Assignment | Mean imbalance | SD | Worst |
|---|---|---|---|
| Simple | −0.00011 | 0.02288 | 0.0821 |
| Stratified | +0.00001 | 0.01009 | 0.0373 |
Both are unbiased: the mean imbalance is essentially zero for both, exactly as promised.
Look at the SD column. Simple randomisation's imbalance scatters more than twice as widely, and its worst draw is more than twice as bad.
You are running one experiment, not four thousand. Unbiasedness is a property of the average over hypothetical repeats. What you get is one draw, and the spread of that draw is what a confidence interval is made of. Stratifying cut it by 56%.
How it works
Split users into strata on something you know before assignment, then randomise within each stratum. Meridian uses quintiles of pre-period trips:
| Stratum | Users | Mean prior trips | Baseline conversion |
|---|---|---|---|
| 0 | 9,089 | 0.00 | 0.1562 |
| 1 | 11,432 | 1.00 | 0.1626 |
| 2 | 10,322 | 2.00 | 0.1767 |
| 3 | 14,558 | 3.43 | 0.1897 |
| 4 | 14,599 | 7.09 | 0.2504 |
Half of each stratum goes to treatment, so the two arms are forced to have the same mix of light and heavy riders. The imbalance that simple randomisation would have produced is removed before the experiment starts rather than adjusted for afterwards.
Blocking is the same idea taken to its limit: form blocks of two matched users and flip a coin inside each. Same mechanism, finest possible strata.