Selection Bias

By the end of this lesson, you should be able to: split any naive comparison into effect and selection bias, say what the selection term is a fact about, work out which direction it runs before you see the data, and state exactly what randomisation removes.

Taking the 9.32 apart

Lesson 1 left three numbers that don't fit together. Alder's adopters ship 9.32 more PRs a week, and the effect on those same teams is 1.18.

The gap between them has a name and, unusually for this subject, an exact formula:

$\underbrace{E[Y \mid D=1] - E[Y \mid D=0]}_{\text{what you measured}} = \underbrace{E[Y(1) - Y(0) \mid D=1]}_{\text{ATT}} + \underbrace{E[Y(0) \mid D=1] - E[Y(0) \mid D=0]}_{\text{selection bias}}$

This is an identity rather than an approximation. It follows from adding and subtracting $E[Y(0) \mid D=1]$, and it holds in every dataset that has ever existed.

Because the effect here was planted, we can compute both terms:

The residual is floating-point dust. The split's exact.

87.4% of what the dashboard shows is selection bias. 12.6% is the assistant.

What the selection term actually is

Look at the middle term again, slowly:

$E[Y(0) \mid D=1] - E[Y(0) \mid D=0]$

Every quantity in it is a $Y(0)$. Nothing in that expression involves the assistant at all. It's comparing the two groups in a world where the product was never built.

The teams that adopted were shipping 71% more before the assistant existed. That gap isn't an effect of anything you did. It's a property of who those teams are, and it sits in your comparison whether or not you go looking for it.