Powering for a 0.2pp success lift (and speeding it up)

Power & sample sizeMedium

Problem. You want to detect a +0.2pp lift in successful-session rate (base 85%) for a Bing experiment. The PM asks how many sessions you need and how long it'll take — and whether you can speed it up.

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← A feature only fires on 3% of queries — how do you not wash out the effect?Which test for a success rate vs queries-per-session? →