Building an Experimentation Platform
By the end of this lesson, you should be able to: explain why per-experiment rigour doesn't aggregate into org-level trust, set an error budget and say what it costs, order a platform roadmap by damage prevented, and describe the readout a platform should generate.
The arithmetic nobody runs
Meridian runs 220 experiments a year and reads about 20 metrics on each. Every test is designed properly, powered honestly and analysed correctly at a nominal 5%.
| Metrics per experiment | Readings a year | Spurious at 5% | Experiments with at least one |
|---|---|---|---|
| 1 | 220 | 11 | 11 |
| 5 | 1,100 | 55 | 51 |
| 20 | 4,400 | 220 | 141 |
| 60 | 13,200 | 660 | 210 |
141 of 220 experiments will contain at least one spurious significant metric. Not because anyone peeked, mis-specified a test or chose a metric after the fact. Because .
This is the gap between being a good experimenter and running a good platform. Individual rigour is necessary and it does not aggregate. The platform's job is everything between the readout and the launch.
Pricing the gates
Meridian's corpus of 220 experiments has a known true effect for each, so every shipping rule can be scored on what it actually produced. Net of the $240,000 a year each shipped change costs to maintain:
| Rule | Shipped | Duds | Net value |
|---|---|---|---|
| Ship anything positive | 120 | 17 | −$17,034,956 |
| + require significance | 41 | 0 | −$2,729,490 |
| + require clearing the ROPE | 23 | 0 | −$1,344,021 |
| + shrink first, then check the ROPE | 4 | 0 | +$192,478 |
| + tighten α to 0.01 as well | 4 | 0 | +$192,478 |
Four things worth noticing.
Significance alone loses $2.7M. It ships 41 changes, every one a genuine improvement, and the small true effects can't cover 41 lots of maintenance. Being right about the sign is not being right about the decision.
Adding a ROPE halves the loss and still loses. Better, not sufficient.
Shrinking before the ROPE check is what flips it positive. Because significant winners overstate by 1.6x, a raw estimate clears the bar when the true effect doesn't. Correcting first removes exactly those.
Tightening α on top changes nothing. The last row ships the same four things. Once the ROPE and shrinkage are in place, α is no longer the binding constraint, which is the opposite of where most governance conversations start.