Power and Sample Size

By the end of this lesson, you should be able to: name the four quantities that decide how many users a test needs, calculate a sample size, explain why small effects need so many users, turn a sample size into a number of days, and say what goes wrong when a test is too small.

Four quantities, one sample size

The sample size of an experiment comes from four things.

The significance level (α). How often you accept a false positive when the change does nothing. Usually 5%.
Power. The chance of detecting a real effect of a given size. Usually 80%, which means a real effect of that size is missed one time in five.
The minimum detectable effect (MDE). The smallest effect you care about detecting. This is a business decision: how big does a change have to be to be worth shipping?
The variance of the metric. How much the metric naturally varies between users. A noisy metric needs more users.

For a conversion rate, the number of users needed in each group is:

n = \frac{(z_{1-\alpha/2} + z_{\text{power}})^2 \times \big[p_1(1-p_1) + p_2(1-p_2)\big]}{(p_2 - p_1)^2}

Here, $p_1$ is the current conversion rate and $p_2$ is the rate you'd see with the smallest effect you care about. With α = 5% and 80% power, the first part, $(1.96 + 0.84)^2$ , is about 7.85.

A handy shortcut for any metric: you need about $16\sigma^2 / \delta^2$ users per group, where σ is the metric's standard deviation and δ is the absolute effect you want to detect.

Why small effects are so expensive

The effect size sits squared at the bottom of the formula. So halving the effect you want to detect means four times as many users, not two.

Try it: how many users a test needs

Current conversion rate5%Smallest lift worth detecting5%Users entering per day20,000

Two groups of equal size, a two-sided test at the 5% level, 80% power. Halving the lift you want to detect roughly quadruples the users you need.

Users per group

122,121

to detect 5% → 5.25%

Users in total

244,241

Days needed

13 days

round up to 2 full weeks

With a 5% conversion rate, detecting a 10% relative lift (5% to 5.5%) needs about 31,000 users per group. Detecting a 5% lift needs about 122,000. Detecting a 2.5% lift needs about 483,000.