Regression Adjustment

By the end of this lesson, you should be able to: rebuild a regression coefficient by hand from residuals, say which of your controls changed the estimate and which only changed the interval, judge in advance whether a covariate is worth adding, and name the kind of control that makes a real gain read as harm.

The regression everyone runs

Alder Cloud shipped an AI coding assistant, and teams turned it on when they felt like it, so this isn't an experiment. For each of 6,000 teams we know team size, how long the team has existed, and its throughput in the quarter before the assistant existed.

The outcome is weekly merged pull requests, logged, so read every coefficient below as a percentage change. The planted truth is a 6.0% lift, which is +5.83% once logged, and that's the number to beat.

Adding four columns did two things at once. The estimate fell by 43.07 points, and the standard error fell by a third. Almost everyone reports the second regression and describes both changes with the same four words, "we controlled for confounders."

They're separate mechanical events with separate consequences, and a regression can hand you one without the other. Saying which one you got is most of what this lesson is for.

Taking the coefficient apart

There's an exact result that splits them. Write the regression you want:

$Y_i = \alpha + \tau D_i + X_i'\beta + \varepsilon_i$

The Frisch-Waugh-Lovell theorem says $\hat{\tau}$, the coefficient on treatment in that multivariate fit, is identical to the slope from this three-step recipe.12

1. Regress treatment $D$ on the covariates $X$. Keep the residuals, $\tilde{D}$.
2. Regress outcome $Y$ on the covariates $X$. Keep the residuals, $\tilde{Y}$.
3. Regress $\tilde{Y}$ on $\tilde{D}$. That slope is $\hat{\tau}$.

Identical is meant literally. Here's each step on Alder's data, with the library's answer at the bottom.

The bottom three agree to $2.61 \times 10^{-15}$, which is floating-point dust rather than a close result.