Why a per-merchant ads A/B is biased — the auction problem

InterferenceHard

Problem. To measure an ads feature, someone proposes randomizing merchants into treatment (new ads treatment) and control. Why is that biased, and how would you design it instead?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Measure whether ads create incremental orders (not cannibalize organic)Which merchants should get ads? Heterogeneous ROAS →