Prevalence is too rare to move in an A/B test — build a metric that can

Measurement & estimationHard

Problem. A team ships an intervention they expect to modestly reduce exposure to a rare harm. Running a fresh LLM-labeling job per arm per day is too slow and costly, and the base rate is so low that per-arm prevalence CIs overlap and nothing looks significant. Give them a metric that can detect the change.

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Stand up an LLM-as-judge measurement you can actually trustThe harm lives partly off-platform — how do you measure it without fooling yourself? →