Potential Outcomes and the Counterfactual

By the end of this lesson, you should be able to: write a causal claim as a comparison of potential outcomes, say why exactly half the necessary data is always missing, tell ATE, ATT and ATU apart and know which one your decision needs, and say precisely what randomisation does.

"The assistant makes teams 6% faster"

Take that sentence seriously for a moment, because it's less obvious than it sounds.

Six percent faster than what? Than they are now? Than they were last quarter? Than other teams? Each of those is a different claim, and only one of them is causal.

The causal version is: faster than the same team, in the same week, if it had not had the assistant. That team doesn't exist. It's the same team in a world that didn't happen.

Everything in this track is machinery for estimating something about a world that didn't happen.

Writing it down

For every team $i$ there are two numbers:

$Y_i(1) = \text{output if the team has the assistant}$ $Y_i(0) = \text{output if it does not}$

Both are properties of the team. Both exist as facts before anyone decides anything. The individual treatment effect is the difference:

$\tau_i = Y_i(1) - Y_i(0)$

Here's the whole difficulty. You observe:

$Y_i^{\text{obs}} = D_i \, Y_i(1) + (1 - D_i)\, Y_i(0)$

where $D_i$ is 1 if the team adopted. Whichever one happened is the one you see, and the other is gone.

On Alder's data, where we planted the effect and so can print both columns:

Team	Adopted	Y(0)	Y(1)	Effect	What you actually observe
2	yes	4.61	4.89	0.28	Y(1). Y(0) is missing
3	yes	18.38	19.48	1.10	Y(1). Y(0) is missing
4	yes	31.95	33.87	1.92	Y(1). Y(0) is missing
1	no	11.75	12.46	0.71	Y(0). Y(1) is missing
6	no	10.53	11.16	0.63	Y(0). Y(1) is missing
8	no	4.08	4.33	0.25	Y(0). Y(1) is missing

Exactly half of that table is unobservable, for every team, always. This is sometimes called the fundamental problem of causal inference, and the word "fundamental" is doing real work: it isn't a sampling problem, a measurement problem or a budget problem. More data adds rows. Every new row arrives with one of its two cells already missing.

Left, the half of the data that does not exist. Middle, three answers to one question. Right, why they differ.

So an individual effect is never knowable. Averages, it turns out, sometimes are.