Data Quality: Doubt the Instrument First

By the end of this lesson, you should be able to: recognise the shapes that mean a pipeline broke rather than behaviour changed, run the internal-consistency check that catches silent logging loss, explain why growth hides instrumentation bugs, and know which checks belong at the start of every investigation.

The metric went up, and it was badly wrong

On 6 May an SDK change stopped Basket's iOS 8.2 build emitting its search event. Users kept searching. They kept adding things to carts, kept checking out, kept ordering. Only the log was wrong.

22,549 search events vanished, 9% of all searches in the window. Here's what the dashboard showed:

Nobody investigated, because the number went up.

That's the part worth sitting with. Every instinct about broken data assumes a cliff: a metric falls off a table and somebody notices by lunchtime. In a growing business the cliff gets filled in by growth, and a catastrophic instrumentation failure shows up as mild underperformance. Searches grew 4.6% when they should have grown 44.9%, which reads as a soft quarter for search rather than as a broken build.

The check that catches it: internal consistency

Segmenting by platform and version would find this eventually, but you'd have to suspect something first, and nothing here looks suspicious. The check that fires without suspicion is different and takes one query.

A funnel has an order. Later steps depend on earlier ones. So the steps have to move together, and when they don't, one of them is lying.

Here's every step's change after 6 May, as logged:

Read it and the impossibility is obvious. Add-to-cart grew 44.6%. You can't add an item to a cart without searching for it. So either 44.6% more people found products without ever running a search, or the searches happened and weren't recorded.

That's a contradiction the data can't resolve on its own, and it doesn't require you to suspect anything in advance. You just need to know the funnel's order and check that reality respects it.

Confirming it takes one more cut:

One cell, missing nearly three quarters of its events. Four cells untouched. A defect that's exactly zero everywhere except one build isn't user behaviour. People don't coordinate.