Your LLM labeler agrees with humans 95% of the time. Is that good?
Measurement & estimationHardProblem. You replaced most human labeling with a multimodal LLM governed by policy prompts. On a gold set, the LLM agrees with subject matter experts 95% of the time. A director says that is excellent and wants to ship the daily prevalence number to leadership. What do you say?
Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.