Measure label quality when there's no ground truth

Measurement & estimationHard

Problem. You need to quantify the quality of a labeling project, but there's no answer key. A colleague suggests "just report the percent of items annotators agree on." Why is that risky, and what do you use?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Testing a new guideline — what's the randomization unit?Quality rose after a tool rollout — but annotators were also learning →