The Back-Door Criterion
By the end of this lesson, you should be able to: apply the back-door criterion to pick an adjustment set, name the three ways a control variable ruins an estimate, tell a precision covariate from a confounder before you fit anything, and say why a well-measured confounder is not the same as a well-adjusted one.
Five columns, all of them tempting
Alder Stream shipped a new recommender. It went out to accounts on the newest app build first, which isn't random, and the question is whether it raised days active over the next 30. Here's the table an analyst has, over 60,000 accounts, 42% of them on the new recommender.
| Column | Correlation with treatment | Correlation with outcome |
|---|---|---|
| Prior weekly hours | 0.383 | 0.678 |
| Titles started | 0.439 | 0.630 |
| Rated a title | 0.231 | 0.326 |
| Household size | -0.001 | 0.230 |
Three of the four correlate with both, which is exactly what a confounder looks like from the outside. Household size correlates with the outcome only, the signature of something that can't be confounding anything.
That table is all the guidance the data provides, and it isn't enough. Here's the graph the data was actually generated from.
The planted total effect is +3.92%, of which +0.40% is the direct arrow and +3.52% travels through titles started.
Six adjustment sets, one dataset
| Controls | Estimate | Off by |
|---|---|---|
| Nothing | +34.45% | 30.53 |
| Prior hours (the back door) | +3.90% | 0.02 |
| Prior hours and household size | +4.01% | 0.09 |
| Prior hours and titles started | +0.54% | 3.38 |
| Prior hours and rated | -3.52% | 7.44 |
| Everything on the table | -6.86% | 10.78 |
| The truth | +3.92% |
Six regressions any of us could defend in a review, four of them wrong, two of those wrong in sign. Every one has a tight standard error and a plausible-looking output.