Propensity Scores
By the end of this lesson, you should be able to: explain what a propensity score is and why it works, estimate and use one, check that treated and untreated users overlap and are balanced, and say what a propensity score can't fix.
What the score is
A propensity score is the probability that a user gets the treatment, given their characteristics:
If engaged, long-tenured users on iOS tend to adopt a feature, their propensity scores are high. Users unlike them have low scores.
The key result comes from Rosenbaum and Rubin in 1983. If a set of covariates is enough to remove confounding (it blocks every back-door path, lesson 4), then the propensity score built from those covariates is also enough. Two users with the same propensity score are, on average, alike on all those covariates, whether or not they were treated.
That turns many variables into one. Instead of finding treated and untreated users who match on age, tenure, platform, country and past activity at once, you compare users with similar propensity scores.
Estimating it
The propensity score is a model of who gets treated, not of the outcome. You fit it with logistic regression, or a machine learning classifier, predicting treatment from the covariates measured before treatment.
Two things to keep in mind:
- The goal isn't prediction accuracy. A model that predicts treatment perfectly is a warning sign, not a success: it means some users were certain to be treated and others certain not to be, so there's nobody to compare them with.
- Only use covariates measured before treatment. Anything the treatment affects doesn't belong in the model (lesson 4).