Multiple Testing: FWER and FDR

By the end of this lesson, you should be able to: explain why testing many metrics or segments produces false wins, choose between controlling the chance of any false win and the share of wins that are false, apply the Bonferroni and Benjamini-Hochberg corrections, and set up a readout so its results can be trusted.

Many metrics, many chances

A 5% significance level means each metric has a 5% chance of a false positive when the change does nothing. One metric, one chance. But experiments rarely track one metric. A typical readout has dozens.

If each of 20 metrics has a 5% chance of a false positive, the chance that at least one of them shows a false "win" is about 64%. Airbnb did this calculation for its guardrail system: with 50 metrics each tested at 5%, an experiment that changes nothing would trigger at least one false alert 92% of the time.

This isn't bad luck. It's arithmetic. Test enough things and something will always look significant.

The same applies to everything else you can multiply:

Metrics: 50 metrics is 50 tests.
Segments: splitting results by 10 countries and 3 platforms is 30 more tests per metric.
Variants: an A/B/C/D test has three comparisons against control.
Time windows: looking at week 1, week 2 and the full period is three tests.

Two things you might want to control

There are two different questions you can ask, and they lead to different corrections.

"What's the chance of any false win at all?" This is the family-wise error rate. Controlling it at 5% means that across all your tests, there's only a 5% chance that even one false positive slips through. Use it when a single false claim is costly, for example when a significant result will be acted on without further checks.

"What share of my wins are false?" This is the false discovery rate. Controlling it at 5% means that, on average, no more than 5% of the results you call significant are false. Use it when you're scanning many metrics to find leads, and you can tolerate a few false ones among many real ones.