Meta-Analysis Across Experiments

By the end of this lesson, you should be able to: estimate the distribution of true effects from a corpus of past results, explain the winner's curse and why small studies suffer it worst, shrink a result using an empirical prior, and use the corpus to size the next experiment.

Two years of experiments

Meridian has run 220 experiments. Because this is a generated corpus, the true effect of every one is known, which is the only way to score what a company's process actually produces:

More than half the changes genuinely helped, and fewer than a fifth were detected. That's not a broken platform, it's Type II error at portfolio scale: most real effects are small, and most experiments aren't powered for small.

The corpus knows its own prior

Here's the move that makes a corpus more than an archive.

The spread of observed results is the spread of true effects plus the sampling noise. Those add, so subtracting gives you the distribution of true effects using nothing but published numbers:

$\operatorname{Var}(\text{observed}) = \tau^2 + \overline{SE^2} \quad\Longrightarrow\quad \hat\tau^2 = \operatorname{Var}(\text{observed}) - \overline{SE^2}$

On Meridian's corpus:

The corpus recovers the distribution it was drawn from. No single experiment can do this, because a single experiment cannot separate its own noise from the spread of effects it was drawn from. Two hundred of them can.

That estimated distribution is a prior, fitted from data rather than chosen. It's what "empirical Bayes" means, and everything below is an application of it.