Data Quality: Doubt the Instrument First

By the end of this lesson, you should be able to: tell whether a number changed because users changed or because the measurement changed, spot the patterns that point to a broken pipeline, and run the checks that catch data problems even when nothing looks wrong.

A metric is the end of a chain

Every number on a dashboard is the last step in a chain.

1. A user does something, like tapping "search".
2. The app or the server records an event.
3. The event is sent, stored and processed by a data pipeline.
4. A query counts the events and turns them into a metric, using some definition.

A problem in any link changes the number, even if nothing about the users changed. An app release that stops sending one event, a browser that drops events when the page closes, a pipeline job that runs twice, a definition that counts the wrong thing: all of them look exactly like a change in user behavior.

So when a metric moves, there are always two possible explanations: the users changed, or the measurement changed. Rule out the second before you explain the first.

When a good number is wrong

People check data carefully when a number falls. When it rises, they celebrate. That's backwards, and Microsoft's experimentation team learned it the hard way.

They call it Twyman's law: "Any figure that looks interesting or different is usually wrong."1 They write that most of the time a surprisingly good result turns out to be a bug, often in how the data was recorded.

Two examples from Bing show how.

More clicks from a slower page. Bing added some JavaScript to its search results page. Extra code usually slows a page down, so the team expected clicks to dip slightly. Instead, users seemed to click more. The real reason: some browsers drop the small tracking call that records a click when the user is leaving the page. The new script happened to make that tracking more reliable. Users weren't clicking more. More of their clicks were being recorded.

Months of gains that were partly tracking. Over several months in 2013, Bing moved its traffic to a new content delivery network. During that time, teams reported their metrics improving: more clicks on the home page, more use of features, fewer people giving up. Much of that turned out to be the new network recording clicks more reliably. When they measured it, click loss for some browsers had dropped by more than 60%. A large part of the gains over that time were in the measurement, not the product.

Neither error made a number look broken. Both made numbers look better. That's why "check the data" can't be something you only do when a number looks bad.