Sample Ratio Mismatch

By the end of this lesson, you should be able to: run a sample ratio test and say precisely what a failure does and does not tell you, explain why demographic balance checks can pass on a badly broken experiment, check pre-period balance on the right variable, and turn down a result that looks like a win.

A result you should not ship

Basket's discovery team shipped a recommendation shelf to half its users for two weeks. Here is the readout.

The 95% confidence interval on that lift runs from +4.97% to +11.34%. It doesn't contain zero, it isn't marginal, and on a conversion metric an eight percent relative gain is a very good quarter.

The true effect of that shelf was +5.00%.

Nobody faked anything and the arithmetic above is right. The experiment measured something 1.63 times larger than what it shipped, and every clue you needed was sitting in the first column of that table.

Count the users before you read the metric

Both arms were meant to get half the traffic. Control got 36,625 and treatment got 34,161, so treatment's share came out at 48.26%. That is close enough to fifty that most people scroll past it. Don't. Test it.

For a design that splits traffic evenly, the chi-square statistic collapses to something you can do in your head:

$\chi^2 = \frac{(n_c - n_t)^2}{n_c + n_t}$

where $n_c$ and $n_t$ are the user counts in control and treatment. The full goodness-of-fit formula compares each arm against its expected count and sums the squared errors, and when the expectation is an even split it collapses to exactly this.

Put the numbers in. The gap is 2,464 users on a total of 70,786:

$\chi^2 = \frac{2{,}464^2}{70{,}786} = 85.77$

On one degree of freedom that gives $p = 2.0 \times 10^{-20}$. For comparison, Basket's checkout redesign ran the same design in the same fortnight and came out at 41,813 against 42,187, which is $\chi^2 = 1.67$ and $p = 0.20$. That's what a working randomiser looks like.

A p-value of $10^{-20}$ isn't a close call. Flip a fair coin 70,786 times and you'd essentially never see a gap this size. Something removed users from one arm and not the other.