Matching and Inverse-Probability Weighting
By the end of this lesson, you should be able to: estimate an effect by matching and by inverse-probability weighting, say which average effect each one estimates, spot extreme weights and deal with them, and explain why good balance doesn't guarantee a right answer.
A small example to work through
Suppose a product has 2,000 users: 1,000 heavy users and 1,000 light users. Heavy users are much more likely to try a new feature.
| Segment | Users | Share who use the feature | Outcome without the feature | Outcome with the feature |
|---|---|---|---|---|
| Heavy | 1,000 | 80% | 10 | 11 |
| Light | 1,000 | 20% | 2 | 3 |
The feature adds exactly 1 for everyone. Now compare users and non-users directly.
- Users: 800 heavy (outcome 11) and 200 light (outcome 3). Average: 9.4.
- Non-users: 200 heavy (outcome 10) and 800 light (outcome 2). Average: 3.6.
The naive difference is 5.8, almost six times the true effect, because the users are mostly heavy users and the non-users mostly light ones. The propensity score here is simple: 0.8 for heavy users and 0.2 for light users.
Matching
Matching pairs each treated user with an untreated user who has the same, or a very similar, propensity score. Then it compares the pairs.
In the example, each heavy user of the feature is matched with a heavy non-user, and each light user with a light non-user. Heavy pairs differ by 11 − 10 = 1. Light pairs differ by 3 − 2 = 1. The matched estimate is 1.
Some things to know about matching:
- It usually estimates the effect on the treated (ATT), because you start from the treated users and find comparisons for them.
- Untreated users with no match are dropped. That's fine, but it means the comparison only covers the kinds of users who appear in both groups.
- A caliper sets how close a match must be. A treated user with no untreated user within the caliper gets no match, and is dropped. Report how many were dropped.