Potential Outcomes and the Counterfactual

By the end of this lesson, you should be able to: describe a causal effect using potential outcomes, explain why an individual effect can never be observed directly, tell apart the average effect for everyone, for the treated and for the untreated, and explain how randomisation lets you estimate an average you can't see.

Two outcomes for every person

Does an AI coding assistant make developers faster? To answer that for one developer, you'd need to know two things:

how long they take with the assistant, and
how long they'd take without it.

These are called potential outcomes. Write them as $Y(1)$ , the outcome with the treatment, and $Y(0)$ , the outcome without it. The causal effect for that person is the difference:

\text{Effect for one person} = Y(1) - Y(0)

The problem is that each person only lives one of the two versions. A developer who uses the assistant gives you $Y(1)$ . Their $Y(0)$ , how long they'd have taken without it, never happens. It's called the counterfactual, and it's missing for everyone, always.

This is sometimes called the fundamental problem of causal inference. You can never observe an individual causal effect. Everything in this track is a way of estimating averages of effects you can't see.

A real experiment: GitHub Copilot

In 2022, researchers from Microsoft Research and GitHub ran a randomised experiment on GitHub Copilot, an AI assistant that suggests code. They recruited 95 professional developers and asked them to write a web server in JavaScript as quickly as possible. A coin flip decided who got Copilot: 45 did and 50 didn't.

Among the developers who finished, those with Copilot took an average of 71 minutes. Those without took 161 minutes. Copilot users were 55.8% faster, with a 95% confidence interval from 21% to 89%.

No developer did the task both ways. Every person in the treatment group is missing their "without Copilot" time, and every person in the control group is missing their "with Copilot" time. Yet the experiment can still estimate the average effect. Why that works is the rest of this lesson.