A/B two matching models — but a person's devices split across arms

InterferenceHard

Problem. You want to A/B two matching models on live traffic. But your randomization unit is the device, and a single person's devices can land in different arms. Is the test valid? What's the right unit?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Your match model has AUC 0.95 but ships badly — why calibration?A consent change hit coverage — measure its true impact →