Meta-Analysis Across Experiments

By the end of this lesson, you should be able to: estimate the spread of true effects from your past results, explain the winner's curse and when it's worst, shrink a result toward what your past experiments suggest, and use them to size the next test.

What your past experiments know

A company that runs experiments builds up a record: hundreds of tests, each with an estimated effect and a standard error. Each one, alone, says something about one change. Together, they say something no single test can: what effects in this product usually look like.

That matters because most ideas don't do much. Microsoft's experimentation team reported in 2009 that "only about 1/3 of ideas improve the metrics they were designed to improve".3 Most true effects are small, which shapes how every individual result should be read.

The record knows the spread of true effects

Each observed result is the true effect plus sampling noise. Across many experiments, their variances add:

$\operatorname{Var}(\text{observed}) = \tau^2 + \overline{SE^2} \quad\Longrightarrow\quad \hat\tau^2 = \operatorname{Var}(\text{observed}) - \overline{SE^2}$

Here $\tau$ is the standard deviation of the true effects, and $\overline{SE^2}$ is the average squared standard error, which every experiment reports. Subtract one from the other and you've estimated how big true effects typically are, using only published numbers. No single experiment can do this, because it can't separate its own noise from the spread it came from. A few hundred can.

That estimated distribution is a prior, fitted from data rather than chosen by hand. That's what empirical Bayes means, and everything below uses it.