Randomisation and What an Experiment Proves

By the end of this lesson, you should be able to: explain what an A/B test is and why companies run them, say why before-and-after comparisons and comparisons of users who chose a feature give the wrong answer, explain what random assignment fixes, and state what an experiment does and doesn't prove.

What an experiment is

An A/B test, or controlled experiment, works like this.

1. Take the users who would see a change.
2. Split them at random into two groups.
3. Show one group the change (the treatment) and the other the current product (the control).
4. Compare the two groups on the metric you care about.

Because the split is random, the only systematic difference between the groups is the change. So any difference in the metric, beyond what chance would produce, was caused by the change.

Why companies test instead of trusting judgement

Most ideas don't work, and people are bad at guessing which ones will.

Microsoft's experimentation team reported that only about one third of the ideas tested at Microsoft improved the metrics they were designed to improve.2 The rest did nothing or made things worse. Without a test, all of them would have shipped as "improvements".

The reverse happens too. At Bing, someone suggested a small change to how ad headlines were displayed. It was judged low priority and sat in the backlog for more than six months. When an engineer finally tested it, it increased Bing's revenue by 12%, worth more than $100 million a year in the US alone.1 Nobody's judgement had picked it out. The test did.