Quality rose after a tool rollout — but annotators were also learning

Causal inferenceHard

Problem. A new annotation tool was rolled out to everyone at once. Quality (gold accuracy) rose from 89% to 93% over the following month. Leadership credits the tool. But annotators are always improving with experience. Did the tool actually work?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Measure label quality when there's no ground truthForecast annotation throughput and capacity for planning →