Heterogeneous Treatment Effects (CATE)

By the end of this article, you should be able to: tell the average effect from the effect for a group, explain why slicing an experiment finds differences that aren't there, estimate group effects honestly, and turn them into a decision about who gets the treatment.

The average can hide who it helps

An experiment's headline result is the average treatment effect (ATE): how much the change moved the metric, averaged over everyone. It answers "should everyone get this?"

Often the better question is "who should get this?" A promotion might win back lapsed customers and only give a discount to people who'd have ordered anyway. A new feature might help power users and confuse new ones. The answer to that question is the conditional average treatment effect (CATE): the average effect for users with particular characteristics $x$,

$\tau(x) = \mathbb{E}\left[\,Y(1) - Y(0) \mid X = x\,\right]$

where $Y(1)$ is a user's outcome with the treatment, $Y(0)$ is their outcome without it, and $X$ is what you know about them before the experiment: country, tenure, device, past activity. The ATE is just $\tau(x)$ averaged over everyone. So an ATE near zero can come from a change with no effect, or from one that helps half the users and hurts the other half.

That makes CATE worth estimating. It's also the easiest thing in experimentation to get wrong.

Slicing an experiment finds differences that aren't there

ISIS-2 was a heart-attack trial with 17,187 patients. Aspirin's benefit was overwhelming, with P below 0.00001. The investigators then split the patients by astrological star sign, using birth dates they'd collected anyway. In two of the twelve signs, Gemini and Libra, aspirin appeared slightly harmful.1

Nobody believes aspirin works differently on Geminis. That's the point Peter Sleight, one of the investigators, made with it: "most physicians (but not all!) laughed" at the star signs. But when the same pattern turns up in a grouping that sounds plausible, people find an explanation and believe it. In ISIS-2 itself, one of the 16 countries showed aspirin as slightly negative, and the only explanation was chance.

The explorer starts with an experiment where every segment truly gains exactly +1 point, split into 12 segments like the star signs. Across 500 simulated experiments:

• The best-looking segment measures +2.01 points, double its true effect. The largest of twelve noisy estimates is large because of its noise.
• In 48% of experiments, at least one segment looks harmful, even though every segment truly gains.
• Raise the segments to 24 and the best one measures +2.70, while 96% of experiments show a segment going the wrong way.

Each segment has fewer users than the whole experiment, so its estimate is noisier. Looking at many noisy estimates and picking the extreme one guarantees a dramatic result. It's the same arithmetic as multiple testing, and the same winner's curse as in meta-analysis.