Stand up an LLM-as-judge measurement you can actually trust

Measurement & estimationMedium

Problem. You've decided to use an LLM to label sampled content for a policy. Walk through how you'd stand this up so the labels are trustworthy at scale, and keep them trustworthy as models and policies change.

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Your LLM labeler isn't perfect — how does that bias the number, and how do you fix it?Prevalence is too rare to move in an A/B test — build a metric that can →