A/A Tests and Platform Validation
By the end of this lesson, you should be able to: explain what an A/A test is and what it's for, describe what a healthy set of A/A results looks like, read the shape of a p-value distribution, and say when to run A/A tests.
What an A/A test is
An A/A test runs the whole experiment machinery, splitting users into two groups, logging their behaviour and computing the results, but gives both groups exactly the same product.
Since nothing is different, there's no real effect to find. Any "significant" difference is either chance or a problem in the machinery.
Microsoft's 2009 guide gives A/A tests two jobs.
- Measure how much your metrics vary, so you can plan sample sizes (lesson 5).
- Test the experimentation system. At a 5% significance level, an A/A test should come back significant about 5% of the time. Much more than that means something is broken.
Three years later, Microsoft's team called the A/A test "our most useful tool in identifying issues in practical systems", and recommended that every platform run them continuously.
What a healthy platform looks like
One A/A test tells you very little. A working platform gives a non-significant result 95% of the time, and so does a broken one, much of the time. What tells you something is many A/A tests together.
Run hundreds or thousands of them and look at two things.
The share that come back significant. It should be close to 5%.
The full spread of p-values. It should be flat: about as many p-values between 0 and 0.05 as between 0.05 and 0.10, and so on up to 1. That isn't a convention. When there's no real effect, a p-value is equally likely to land anywhere between 0 and 1. That's what a p-value means.
With a sound analysis, every bin holds about 50 of the 1,000 tests. Now drag the slider to make the analysis underestimate the variance, which is exactly what happens when you analyse sessions as if they were independent users (lesson 3). Small p-values pile up on the left, and the share of significant results climbs well past 5%. At ×2, about one test in six comes up "significant" with nothing different.