More sessions won't help: power when businesses are the bottleneck

Power & sample sizeHard

Problem. The checkout prototype from the pilot is running with buyers randomized inside each of 18 businesses. The business case needs a lift of at least 1.5 points in checkout conversion. The PM asks how many more weeks the test needs to detect that. How do you answer?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Only 18 pilot businesses: how do you design the test?Revenue per checkout session is dominated by one business. Which test? →