Heterogeneous Treatment Effects (CATE)
By the end of this lesson, you should be able to: say what a CATE is and how it differs from the average effect, describe the S-, T- and X-learners, test whether a model's estimated differences are real, and judge whether targeting is worth more than giving the change to everyone.
One effect per person
An experiment gives you an average treatment effect. For many decisions, that's enough. For others, the question is who the effect is biggest for: who should get the discount, the email, the new feature.
The conditional average treatment effect (CATE) answers that. It's the average effect for people with given characteristics :
Instead of one number, you get a function: an estimated effect for each type of user.
Segment analysis, splitting the results by a few groups you chose in advance, is the simplest version (Experimentation lesson 19). eBay found its search ads worked for new and occasional buyers and did nothing for frequent ones. CATE models extend this to many characteristics at once, letting a model find the pattern.
A real example: who responds to social pressure
Before Michigan's August 2006 primary election, researchers mailed some households a letter listing whether each person in the household, and their neighbours, had voted in past elections, and saying an updated list would follow after the election. Households were picked at random: 38,218 people received the "neighbours" letter and 191,243 were in the control group. Turnout rose by 8.1 percentage points.
Künzel and colleagues, who developed the X-learner, re-analysed this experiment to estimate the effect for each voter based on age, gender and voting history.
- The effect was largest for people who had voted in three of the past five elections. That group was also large. With a limited budget for letters, they're the ones to target.
- A small number of people who had voted only once showed a significant negative effect. The letter put them off. Other researchers had reported complaints from recipients to election offices.
The average of 8.1 points hid both a group to prioritise and a group to leave out.