Sample Ratio Mismatch
By the end of this lesson, you should be able to: explain what a sample ratio mismatch is and why it invalidates a result, test for one, name where they come from, and decide what to do when you find one.
A loss that was really a win
A team at MSN tested a bigger carousel on its home page: 16 rotating cards instead of 12. They expected more clicks, because related experiments had shown that. Instead, the experiment showed a significant drop in engagement with the cards. It looked like the change should not ship.
But the platform had raised a warning. The treatment group had fewer users in the analysis than it should have. The test was set up as a 50/50 split, and the gap was far too big to be chance.
The cause turned up in the data processing. Microsoft filters out automated traffic by flagging users whose activity looks too high to be human. Some users in the treatment liked the bigger carousel so much that they clicked past that threshold, and were removed as bots. So the treatment group had lost its most engaged users, and its average engagement fell.
Once the bot filtering was corrected, the result flipped. The 16-card carousel significantly increased engagement, and it shipped.
What a sample ratio mismatch is
A sample ratio mismatch (SRM) is when the number of users in each group differs from the split you configured by more than chance allows.
Randomisation never gives an exact 50/50, so small differences are normal. The question is whether a difference is too big to be luck. You answer that with a chi-square test, which compares the observed counts with the expected ones.
Here's an example from Microsoft's paper on SRMs. An experiment configured as 50/50 ended with 821,588 users in one group and 815,482 in the other. That's a 50.2/49.8 split, which looks harmless. But with this many users, a gap that size would happen by chance less than once in 500,000 experiments. Try it, or enter your own counts.
Because the check is so sensitive, teams usually flag an SRM only at a very small p-value, for example below 0.0005. A real problem produces a tiny p-value, and a strict threshold avoids false alarms across hundreds of experiments.