Your match model has AUC 0.95 but ships badly — why calibration?

Measurement & estimationHard

Problem. Your probabilistic match model has a great AUC (0.95), but when deployed, thresholding its scores gives the wrong precision and frequency caps misfire. What's going on, and what do you check?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Estimate the graph's precision and recall with no ground truthA/B two matching models — but a person's devices split across arms →