Power and Sample Size

By the end of this lesson, you should be able to: compute the sample size a test needs from four inputs, explain why halving the effect you want to catch quadruples the cost, recognise an underpowered test before you run it rather than after, and answer "how long?" with a number instead of "two weeks".

Two weeks is not an answer

Ask a team how long an experiment should run and you'll usually hear "two weeks". Sometimes there's a reason, usually about covering both weekends. Mostly it's a habit.

Here's what two weeks actually buys on Basket's checkout flow, where 18.97% of users place an order and about 6,000 users a day are available to split.

Two weeks is enough to catch a 5% lift and hopeless for a 2% one. Whether "two weeks" is right depends entirely on a number nobody said out loud: how big an effect you're looking for.

The four things that are in tension

Sample size isn't a free choice. It's the output of four quantities that trade against each other, and fixing any three fixes the fourth.

Baseline rate ($p$). What the metric does today. You don't choose this, you measure it. Basket's is 0.1897.

Minimum detectable effect (MDE). The smallest lift you want to be able to catch. This is a product decision, not a statistical one: it's the smallest change that would alter what you do next. If a 1% lift wouldn't change a single decision, don't pay to detect it.

Significance level ($\alpha$). How often you'll accept a false alarm. Almost always 0.05.

Power ($1 - \beta$). How often you'll catch a real effect of exactly MDE size. Convention is 80%, which means one real effect in five gets missed. That's a much worse deal than most people realise when they accept the default.

Put them together and you get the formula:

$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2p(1-p)}{\delta^2}$

where $n$ is users per arm and $\delta = p \times \text{MDE}$ is the absolute difference you're chasing. The $2$ is there because both arms carry sampling error, and you're measuring a difference between two noisy things.

At $\alpha = 0.05$ and 80% power, $z_{1-\alpha/2} = 1.96$ and $z_{1-\beta} = 0.84$, so the bracket is $2.80^2 = 7.85$. That constant is worth memorising: 7.85 times two-p-one-minus-p, over delta squared. You can do it on a whiteboard.