Encouragement Designs and LATE
By the end of this lesson, you should be able to: tell intention to treat apart from the effect on compliers and say which question each answers, convert one into the other, explain why comparing users with non-users fails even inside an experiment, and design an encouragement strong enough to learn from.
You can offer a feature, but not make people use it
In September 2022, Microsoft ran an experiment on GitHub Copilot, the AI coding assistant, with 1,746 of its developers. Half were picked at random and sent an email inviting them to sign up. The other half heard nothing.
Microsoft could randomise the invitation. It couldn't randomise who used the tool:
- In the first two weeks, only 8.5% of invited developers signed up.
- Microsoft sent two reminder emails, and take-up rose to 42.5%.
- Meanwhile, 0.5% of the control group got access anyway, some because they worked on related tools.
This is an encouragement design: randomise an invitation, an email or a nudge toward the treatment, rather than the treatment itself. It's still a real experiment. It's just an experiment about the invitation.
Intention to treat
The simplest analysis compares the two groups as randomised: everyone invited against everyone not invited, regardless of who used the tool. That's the intention-to-treat (ITT) effect.
ITT needs no assumptions beyond the randomisation. Nobody is dropped or moved between groups.
It answers a specific question: what happens when you send the invitation? If the decision is whether to invite all developers, ITT is the right number, because inviting them is exactly what was tested. It includes the people who ignored the invitation, as a real rollout would.