Propensity Scores

By the end of this lesson, you should be able to: fit and read a propensity score, show that one number balances many covariates, find the region where no comparison exists, and pick the diagnostic that notices when the model is wrong.

What the score is a model of

Lesson 5 adjusted for four covariates directly. This lesson collapses them into one number: the probability a team adopts, given what you can see about it.

$e(X) = P(\text{adopted} = 1 \mid X)$

Fitting it is unremarkable: a logistic regression of adoption on log team size, tenure, log prior PRs and log prior reviews, over the same 6,000 Alder Cloud teams, 38.5% of which adopted.

What matters is that this models the assignment mechanism. Its job is to describe how the assistant actually spread, so its coefficients are claims someone on that team could argue with.

Which is also why you shouldn't read them as the mechanism. Team size comes out negative, and bigger teams plainly adopt more: 33.6% in the smallest quartile against 43.3% in the largest. Prior throughput already carries most of team size, so the negative coefficient says that among teams shipping at the same rate, the bigger one adopts slightly less. That's a conditional association, not a story about size.

Calibration is the property you need

The score gets used as a probability: weights divide by it, strata are cut on it. So the question is whether a team with $e(X) = 0.4$ really does adopt 40% of the time.

Across all ten deciles the largest gap between predicted and actual is 0.018. The score is telling the truth about frequencies, which is what it'll be asked to do.

Hold on to that, because calibration is going to fail the test at the end of this lesson.