Selection Bias
By the end of this lesson, you should be able to: split a naive comparison into the treatment effect and the selection bias, explain where selection bias comes from, predict which way it will push a result, and say what randomisation removes.
Taking a naive comparison apart
Lesson 1 compared developers who chose a coding tool with developers who didn't. The data said the tool saved 105.8 minutes. The true effect for the developers who chose it was only 37.5 minutes. Where did the other 68.3 come from?
Write the naive comparison in potential outcomes:
The middle step adds and subtracts the same thing: the treated group's outcome without the treatment. That splits the naive comparison into two parts.
- The effect on the treated: how much the treatment changed things for the people who got it.
- Selection bias: how different the treated people would have been from the untreated people even without the treatment.
In lesson 1's example, the developers who chose the tool would have taken 115 minutes on average without it, against 183.3 minutes for the others. That 68.3-minute head start is the selection bias. Add it to the real 37.5-minute effect and you get the 105.8 the data showed.
A real example: Facebook ads
Facebook's advertising study measured this directly. For one campaign, Facebook ran a randomised experiment that held some users back from seeing the ads, which gives the true effect. Researchers also computed the naive comparison: users who saw the ads against users who didn't.
| Comparison | Increase in purchases from the ads |
|---|---|
| Users who saw the ads vs users who didn't | +316% |
| Randomised experiment | +73% |
The naive comparison was more than four times the real effect. Most of it was selection: the people who saw the ads were people who were active online during the campaign, and the ads were targeted at people likely to buy. They would have bought more anyway.