Your LLM labeler isn't perfect — how does that bias the number, and how do you fix it?

Measurement & estimationHard

Problem. You measure prevalence by having an LLM label a probability sample. The LLM is good but not perfect — it misses some violations and false-flags some safe content. How does that distort the prevalence estimate, and how do you correct it?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Uniform, PPS, or ML-assisted sampling — and what does it cost you?Stand up an LLM-as-judge measurement you can actually trust →