A/A Tests and Platform Validation

By the end of this lesson, you should be able to: say what an A/A test is for and what it cannot tell you, read a p-value histogram as a diagnostic rather than a curiosity, recognise the two failure shapes it exposes, and argue for running one when somebody objects that it wastes traffic.

The experiment where nothing is different

An A/A test is an experiment you run on purpose with no treatment in it. Split the users, ship both arms exactly the same product, wait, then analyse it as though it were a real test.

You know the answer in advance. The true effect is zero, because there's nothing there to have an effect. So every significant result is a false positive, and you can count them.

This sounds like a waste of traffic, and it's the single most valuable thing you can do to an experimentation platform. Every trust check in lesson 7 asks whether this one experiment was assigned properly. An A/A test asks whether the machine that runs all your experiments works at all.

What a healthy platform looks like

Basket's platform, 4,000 A/A tests, 6,000 users an arm.

That's the headline, and on its own it's a weak check. A rate near 5% is necessary but it isn't sufficient, because it only looks at one thin slice of the distribution. The stronger diagnostic is the shape of all the p-values.

Under a working platform with no effect anywhere, p-values are uniform. Every value between 0 and 1 is equally likely, so 5% land in any 5%-wide band you pick:

That uniformity isn't a convention. It's what a p-value is: the probability of seeing something at least this extreme when nothing is happening. If that definition holds, the values have to spread evenly.

Which makes the histogram a proper instrument. It has a shape you can check, and departures from it point at specific problems.