Multiple Testing: FWER and FDR
By the end of this lesson, you should be able to: compute how many false winners a scorecard will produce before you read it, tell family-wise error control from false discovery control and pick the right one, explain why slicing by segment is far more dangerous than adding metrics, and pre-register in a way that survives contact with a stakeholder.
A result that would have shipped
Basket changed some microcopy on the checkout button. The primary metric didn't move, but the scorecard tracked twenty secondary metrics alongside it, which is a completely normal number.
Here are the six most promising, sorted by p-value.
| Metric | Lift | p |
|---|---|---|
| metric_16 | +2.81% | 0.032 |
| metric_01 | +3.56% | 0.232 |
| metric_11 | +2.95% | 0.255 |
| metric_03 | +3.39% | 0.263 |
| metric_14 | +2.32% | 0.290 |
| metric_05 | −2.02% | 0.309 |
One of them is significant. It's got a plausible effect size, a p-value comfortably under the threshold, and it's the kind of secondary win that gets a change shipped when the primary metric is flat.
Every one of those twenty metrics was generated with no treatment effect whatsoever. There's nothing to find in any of them. metric_16 is noise, and it's noise that looks exactly like a result.
This is arithmetic, not bad luck
If a metric has no effect, a test at calls it significant 5% of the time. That's the deal you signed. Test one metric and you have a 5% chance of a false winner.
Test independent metrics and the chance that at least one fires is:
| Metrics tested | Formula says | Simulation says |
|---|---|---|
| 1 | 5.0% | 5.3% |
| 5 | 22.6% | 21.9% |
| 10 | 40.1% | 39.3% |
| 20 | 64.2% | 63.5% |
| 50 | 92.3% | 92.6% |
| 100 | 99.4% | 99.4% |
At twenty metrics the chance of at least one false winner is 64%. Basket's scorecard wasn't unlucky. A scorecard that size produces a spurious winner about two times in three, so the surprising outcome would have been a clean sweep.
This is the same arithmetic as lesson 14, rotated. There you looked at one metric fourteen times. Here you look at twenty metrics once. Both are "how many chances did I give myself to be wrong", and both have the same answer: more than you thought.