Regression Adjustment

By the end of this lesson, you should be able to: explain what the treatment coefficient in a regression estimates, list the assumptions it rests on, explain why adding controls doesn't reliably reduce bias, and tell the safe use of regression in experiments from the risky use in observational data.

The regression everyone runs

When treated and untreated users differ, the usual first move is a regression:

$Y$ is the outcome, $T$ is 1 for treated users and 0 otherwise, and the $X$s are covariates like age, tenure or past activity. The coefficient $\tau$ is read as "the effect of the treatment, controlling for the $X$s".

What it actually does is compare treated and untreated users with the same values of the covariates, using a straight-line model to fill in the comparison where exact matches don't exist.

What the coefficient rests on

For $\tau$ to be the causal effect, three things have to hold.

1. No unmeasured confounding. The covariates block every back-door path between treatment and outcome (lesson 4). Anything that affects both and isn't in the model biases $\tau$.
2. The model is the right shape. The regression assumes each covariate's effect is a straight line and the same for treated and untreated users. If the real relationship curves, or the treatment effect differs by user type, the adjustment is off.
3. Overlap. For each kind of user, there must be both treated and untreated users to compare. Where there aren't, the regression extrapolates its straight line into a region with no data, and nothing warns you.

None of these can be checked from the regression output. A tight standard error and a high R² say nothing about whether they hold.