Experiment Data Quality
By the end of this lesson, you should be able to: tell a data problem that only costs precision from one that biases the result, explain why the change under test so often affects its own measurement, and run the checks that catch these problems before you read any metric.
Two kinds of broken data
Every experiment's data has some problems. Events go missing, pipelines fail for an hour, bots get in. What matters is whether the problem affects the two groups equally or differently.
Symmetric problems hit both groups the same way. Say logging fails for everyone for two hours on a Tuesday. Both groups lose the same share of data, for the same reasons. The comparison between them is still fair. You lose some precision, so the confidence interval is a little wider, but the answer isn't pushed in either direction.
Asymmetric problems hit one group more than the other. Now the groups differ in how they're measured, not just in the product they see. The difference you measure is partly the change and partly the measurement, and you can't separate the two. An asymmetric problem can create an effect that doesn't exist, hide one that does, or reverse its sign.
Symmetric problems are annoying. Asymmetric ones are dangerous.
A slower page that "got more clicks"
Most websites record a click by sending a tiny request to a server, called a beacon, at the moment the user clicks. The trouble is that the click also takes the user to a new page, and many browsers cancel pending requests when you leave a page. So some clicks are never recorded.
How many? Microsoft found that browsers like Chrome, Firefox and Safari often cancelled these requests, and on Safari the loss was sometimes over 50%. Internet Explorer kept sending them, so it lost far fewer.
Then Bing ran an experiment that added a little extra code when a user clicked a search result. The code slowed things down slightly. Yet the experiment showed that users clicked more.
They didn't. The extra code delayed the move to the next page just long enough for more beacons to get through. The treatment wasn't getting more clicks. It was recording more of them. Microsoft's team wrote that they had seen multiple experiments where an added delay made the treatment look artificially better.
An earlier MSN experiment had the same problem. It changed the Hotmail link on the MSN home page to open in a new tab. Users seemed to click it more. But opening a new tab doesn't leave the current page, so the click beacon wasn't cancelled. Most of the "increase" was just more clicks being recorded, in browsers other than Internet Explorer.
Notice that neither experiment had a sample ratio mismatch. The user counts were right. Only the events inside each user's data were recorded differently.