Matching and Inverse-Probability Weighting
By the end of this lesson, you should be able to: estimate and check a propensity score, match and weight with it, read a balance table properly, and explain why a balance table that looks perfect is not evidence that your estimate is right.
A number that would get the tool bought
Alder Cloud rolled out an AI coding assistant. Adoption is voluntary, so some teams turned it on and some didn't.
| Teams that adopted | 20.78 PRs/week |
| Teams that didn't | 11.46 PRs/week |
| Gap | +59.52% |
The true effect is +5.83%. The gap is 10.2 times the effect.
They were different before they adopted anything
Look at what the two groups looked like before the assistant existed.
| Adopters | Others | Standardised difference | |
|---|---|---|---|
| Prior PRs per week | 15.98 | 10.43 | +0.826 |
| Prior reviews per week | 35.52 | 25.38 | +0.691 |
| Team size | 8.51 | 7.82 | +0.142 |
| Tenure, days | 275 | 293 | −0.092 |
The standardised mean difference expresses each gap in standard deviations, so covariates on different scales are comparable. Anything past 0.1 is conventionally a problem, and prior throughput is at 0.83.
Adopters were already shipping 53% more before they adopted. The assistant did not do that.