Experiment Data Quality

By the end of this lesson, you should be able to: tell which data faults matter and which merely cost precision, know why the dangerous ones cluster in the treatment arm, run a six-query pre-analysis check, and explain why the sample ratio test misses most of this.

Two kinds of broken data

Lesson 7 checked that the split was sound. This is a different question: is the data sound? An experiment can be perfectly randomised and still measured with a broken instrument.

The useful distinction isn't between big faults and small ones. It's between faults that hit both arms and faults that don't.

Symmetric faults barely matter

Take Basket's checkout experiment, which really does have a +5.5% effect, and break it in two ways that affect both arms equally.

Nine thousand bots is a tenth of the sample, and the answer moves by three hundredths of a percentage point.

The reason is arithmetic rather than luck. A relative lift is a ratio of two means. A symmetric fault multiplies both means by nearly the same factor, and the factor cancels:

$\frac{k \cdot \bar{Y}_T}{k \cdot \bar{Y}_C} = \frac{\bar{Y}_T}{\bar{Y}_C}$

What you lose is precision. Bots add rows that contain no signal, so the standard error grows and the test gets weaker. That's a real cost and it isn't a wrong answer.