How many accounts do you need for a 0.5pp lift?

Power & sample sizeMedium

Problem. You're A/B testing a checkout change expected to lift payment-success by about 0.5 percentage points off an 80% base. The PM asks: how many accounts do you need, and how long will it run?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

Pin down the setup before solving:

"What's the smallest lift worth shipping — is +0.5pp the business-meaningful MDE, or just a guess?"
"Are we powering on accounts, given the household-interference point, or on profiles?"
"What's the daily eligible checkout traffic, so I can turn a sample size into a runtime?"

Lay out your approach

Pin the MDE to business value. Power to detect the smallest lift worth shipping — say +0.5pp on payment-success off an 80% base — not "any effect."
The four inputs. baseline rate, the MDE, significance $\alpha$ (usually 5%), and power $1-\beta$ (usually 80%) — on the right unit (accounts).
Compute n per arm, then translate to runtime via daily eligible traffic.
Pad for the weekly cycle — run at least one to two weeks to cover seasonality and novelty, even if the raw sample arrives sooner.
Speed it up with CUPED on a pre-period covariate, which cuts the sample you actually need.

Then check in: "So the plan is pin the MDE from business value, plug in baseline / α / power for n per arm, then convert to weeks with daily traffic. Want me to run the numbers?"

Simulate the back-and-forth

Interviewer

Just give me a number — how many accounts?

You

It depends most on the MDE, so let me pin that first: the smallest lift worth the engineering cost, say +0.5pp on an 80% base. With 5% significance and 80% power, the two-proportion formula gives roughly a hundred thousand accounts per arm — about two hundred thousand total.

Interviewer

Say we want to detect a quarter-point instead. Does that change much?

You

A lot — sample scales with one over the MDE squared, so halving the MDE roughly quadruples the sample. A 0.25pp MDE is about four hundred thousand per arm. That's the trade-off to put in front of the PM: a smaller detectable effect costs quadratically more traffic.

Interviewer

We get about 50k eligible accounts a day. So four days and we're done?

You

Traffic-wise yes, 200k over 50k a day is four days — but I'd still run one to two weeks. A four-day window misses the weekly cycle and novelty effects, so the read wouldn't be trustworthy even though the sample count is met.

Interviewer

Any way to get there faster without a bigger MDE?

You

CUPED on pre-period engagement. It cuts the variance, which cuts the sample I need for the same power — the honest way to speed it up, versus just relaxing the MDE or peeking early.

Land a clear conclusion

You

So: pin the MDE to business value, power on accounts, and the two-proportion formula gives ~100k per arm for +0.5pp — run two weeks for the weekly cycle, and use CUPED to get there faster. The MDE is the lever: a quarter-point would cost 4× the traffic.

Key summary — what really matters.

Pin the MDE to business value — it dominates everything, since $n \propto 1/\text{MDE}^2$ .
Four inputs: baseline, MDE, $\alpha$ , power — on the account unit.
~100k per arm for +0.5pp off an 80% base; ~200k total.
Translate to runtime, but run 1–2 weeks for the weekly cycle and novelty.
CUPED buys back sample without touching the MDE.

Type I vs Type II. $\alpha$ = false positive (call nothing a win); $\beta$ = miss a real effect; power $= 1-\beta$ , the chance of catching a true effect of size MDE.

Sample size for two proportions (per arm):

$n \approx \frac{(z_{\alpha/2}+z_\beta)^2\,[\,p_1(1-p_1)+p_2(1-p_2)\,]}{(p_1-p_2)^2}$

where the $z$ values are normal quantiles — $z_{\alpha/2}=1.96$ for 5% two-sided, $z_\beta=0.84$ for 80% power. Plug in $p_1 = 0.80$ , $p_2 = 0.805$ , MDE $= 0.005$ :

$n \approx \frac{(1.96+0.84)^2\,[\,0.8\cdot0.2 + 0.805\cdot0.195\,]}{(0.005)^2} = \frac{7.84 \cdot 0.317}{0.000025} \approx 99{,}500 \text{ per arm}$

— call it ~100k accounts per arm, ~200k total.

The MDE drives everything, since $n \propto 1/\text{MDE}^2$ :

MDE	n per arm
0.25 pp	~400k
0.5 pp	~100k
1.0 pp	~25k

n per arm

Illustrative: required sample per arm vs MDE. Halving the effect you want to detect roughly quadruples the sample — why MDE choice dominates.

Translate to runtime. If ~50k eligible accounts enter the checkout funnel per day, 200k / 50k = 4 days of traffic — but run at least one to two weeks to cover weekly seasonality and let novelty settle. And remember the unit: power on accounts, where clustering inflates variance — CUPED on pre-period engagement reduces the variance by a factor $(1-\theta^2)$ , buying the sample back.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you pin down the causal quantity, the population, and the decision before any math.	You jump to a test or a number without saying what you're estimating or why.	You name the estimand, the unit, the population, and the decision it drives.
Design validity	Whether the design actually identifies the effect — randomization unit, interference, selection/confounding.	You analyze a biased or diluted comparison as if it were clean.	You pick the unit that contains interference, name the threat and its direction, and control it.
Statistical rigor	Whether the stats hold up — power and sample size, variance, the right test, and the assumptions behind them.	You misuse a test, ignore power or variance, or lean on an unchecked assumption.	You size the test, reduce or adjust variance, choose the fitting test, and state what would break it.
Decision & trade-offs	Whether you turn the result into a call — guardrails, cost, and what would flip it.	You stop at "significant / not," or ship on one metric while a guardrail breaks.	You land a ship/hold call, weigh the trade-offs, and name the number that changes it.
Communication	Whether you reason clearly, lead the conversation, and translate the stats for a stakeholder.	You dump jargon, or can't explain the method in plain words.	You reason out loud in order, check in, and give a plain recommendation.

Interview tips

Strong-answer signal: pinning the MDE to business value, tying MDE → sample → runtime, powering on the account unit, and padding for the weekly cycle. Common trap: powering on profiles, picking an MDE out of the air, or stopping at "4 days of traffic" and missing weekly seasonality.