What an A/B test is

An A/B test is a simple experiment. You split users at random into two groups: group A (the control) sees the current version, and group B (the treatment) sees the new version. Because random splitting makes the two groups alike in every other way, any difference in their behavior is caused by the one thing you changed. That is the whole point — an A/B test turns "we think this is better" into "we measured that this is better."

We'll follow one running example. An online store, ShopFast, wants to test a new green "Buy now" button. The current grey button is the control; the green button is the treatment. The number they care about is checkout conversion — the share of visitors who finish a purchase.

The framework: six steps

Run every test through the same six steps. In an interview, simply saying these in order already puts you ahead of most candidates.

1. Start with a clear hypothesis

Write one sentence: "Changing X will improve Y, because Z."

ShopFast: "Making the Buy button green and bigger will raise checkout conversion, because it stands out and is easier to tap on a phone."

A good hypothesis names the change, the metric, and a reason. If you cannot write it, you are not ready to test.

That plain-English sentence is really making a statistical claim, and it's worth writing that claim out precisely before you touch any data. Every test compares two hypotheses:

• Null hypothesis, $H_0$: the change makes no real difference. Any gap you observe is just noise. For ShopFast, that's $H_0: p_{\text{treatment}} = p_{\text{control}}$, both buttons convert at the same true rate, say 5%.
• Alternative hypothesis, $H_1$: the change does make a difference. For ShopFast, that's $H_1: p_{\text{treatment}} \neq p_{\text{control}}$.

Notice $H_1$ says "different," not "better." Even though ShopFast's hunch is that green converts higher, the test should stay two-sided ($\neq$, not just $>$), so it can also catch the uncomfortable case where the new button quietly makes things worse. You start the test assuming $H_0$ is true, and only abandon it if the data makes that assumption hard to believe.

What could go wrong: Type I and Type II error

Once you have $H_0$ and $H_1$, there are exactly two ways to get the call wrong:

• Type I error ($\alpha$): you ship a button that does nothing, believing it works. ShopFast fixes $\alpha = 0.05$, so at most a 5% chance of this happening if the button truly does nothing.
• Type II error ($\beta$): the button genuinely helps, but the test fails to catch it, so you throw away a real win. ShopFast targets $\beta = 0.20$, i.e. 80% power ($1 - \beta$), an 80% chance of detecting the effect if it's really there.

This is exactly where the $z_{\alpha/2}$ and $z_\beta$ in the sample-size formula below come from: $z_{\alpha/2} = 1.96$ is the cutoff that keeps your false-positive rate at 5%, and $z_\beta = 0.84$ is the cutoff that keeps your false-negative rate at 20%. Tightening one, say demanding $\alpha = 0.01$ instead of 0.05, makes the other harder to hold without collecting more data; that tension is why sample size calculations exist at all.

2. Pick your metrics

You need three kinds:

• Primary metric — the one number that decides success. (ShopFast: checkout conversion.) Pick a single one. If you track five and one happens to look good, you will fool yourself.
• Guardrail metrics — things that must not get worse. (ShopFast: average order value, refund rate, page-load time.) A change can win on the headline number and quietly hurt the business elsewhere.
• Supporting metrics — extra context that helps explain why the number moved.

Avoid "vanity" metrics like clicks on the new button. Of course people click a bigger button — that does not mean they buy more. Measure the real outcome, not the feature.

3. Choose the randomization unit

Decide what you split. Usually it is the user, not the visit, so the same person always sees the same version (otherwise they get a confusing mix).

Pick a unit big enough to contain "spillover." If one user's treatment can affect another — shared accounts, group chats, a marketplace where buyers and sellers meet — split by the bigger unit, like the household or the city, even though it costs you some statistical power. A lopsided split between the two groups is a red flag that the setup is broken, so check it first; this is called a sample ratio mismatch.

4. Work out the sample size and run time

Before launching, answer two questions: how many users do we need, and for how long?

The sample size depends on three things:

• The baseline rate (ShopFast starts at 5% conversion).
• The smallest lift worth detecting, the MDE, or minimum detectable effect (say, +0.5 percentage points).
• How sure you want to be, usually 80% power and a 5% false-positive rate.

There's an actual formula behind "smaller effects need more users," and it's worth knowing by name even if you'd never compute it by hand in an interview:

$n \approx \frac{(z_{\alpha/2}+z_\beta)^2\,[\,p_1(1-p_1)+p_2(1-p_2)\,]}{(p_1-p_2)^2}$

where $p_1$ and $p_2$ are the baseline and target conversion rates, and $z_{\alpha/2}$, $z_\beta$ are the Type I and Type II error cutoffs from step 1 (for the usual 5% significance / 80% power choice, $z_{\alpha/2} = 1.96$ and $z_\beta = 0.84$). In plain words: this formula asks how many users it takes before the gap you're hoping to see, $p_1 - p_2$, clearly outgrows the natural noise in a proportion, and since that noise shrinks only as $1/\sqrt{n}$, closing in on a small gap costs you the square of however much smaller the gap gets. That squaring is the whole reason effect size dominates the calculation.

There's a mental-math shortcut worth carrying into any interview: at the usual 5% significance and 80% power, $(z_{\alpha/2}+z_\beta)^2 \approx 7.84$, and since both arms contribute roughly the same variance, $p(1-p)$, doubling that gives almost exactly 16. So the whole formula collapses to:

$n \approx \frac{16\, p(1-p)}{(p_1 - p_2)^2}$

using just the baseline rate p and the absolute gap you want to catch. For ShopFast, $16 \times 0.05 \times 0.95 / (0.005)^2 \approx 30{,}400$, close enough to the fuller calculation below to size a test on the back of an envelope, with no calculator needed beyond the baseline rate and the MDE.

The "16" isn't specific to proportions, it's just $2(z_{\alpha/2}+z_\beta)^2$ rounded, so the same shortcut works for any metric once you swap in its variance:

$n \approx \frac{16\,\sigma^2}{\Delta^2}$

where $\sigma^2$ is the metric's variance and $\Delta$ is the smallest difference in means worth catching. (A proportion is just the special case where $\sigma^2 = p(1-p)$.)

Worked example on a different metric. Say ShopFast also wants to know if its planned test has enough power to catch a move in its guardrail, average order value, which has a standard deviation of about $25. They'd only want to react to a genuine $2 shift either way:

$n \approx \frac{16 \times 25^2}{2^2} = \frac{16 \times 625}{4} = 2{,}500 \text{ per arm}$

Just 2,500 per arm to catch a $2 move in AOV, far fewer than the 31,000 per arm the test is already sized for on conversion. That's a common pattern: the primary metric usually dictates the sample size, and guardrails with reasonably large expected moves ride along for free once you've sized for the metric you actually care about most.

Plug the fuller version into ShopFast's exact numbers, $p_1 = 0.05$, $p_2 = 0.055$:

$n \approx \frac{(1.96+0.84)^2\,[\,0.05 \times 0.95 + 0.055 \times 0.945\,]}{(0.005)^2} \approx 31{,}000 \text{ per arm}$

Roughly 31,000 visitors per arm, about 62,000 total. The MDE dominates this number, since n scales with 1/MDE², so halving the effect you want to catch roughly quadruples the sample you need:

If ShopFast's checkout funnel sees about 3,000 visitors a day, 62,000 total is roughly three weeks of traffic. Even if the raw number would arrive sooner, run for at least one or two full weeks regardless, so you cover both weekday and weekend behavior. Behavior on a Tuesday is not behavior on a Sunday.

Quick check. If the PM says a 0.5pp lift is nice but they'd really only act on a full 1pp lift, does the test get easier or harder to run, and by roughly how much?

5. Run it — and don't peek

Launch the test and let it reach the planned size. The hard rule: do not stop early just because the numbers look exciting. Early results bounce around a lot. If you keep checking and stop at the first good moment, you will call far more fake winners than the 5% you signed up for.

If you genuinely need to watch a test while it runs, use a method built for that — sequential testing — instead of peeking at a normal fixed-size test.

6. Read the results and decide

Three weeks later, ShopFast has reached its planned sample size. Compare the two groups.

The green button converted at 5.6% versus 5.0% — a +0.6 point lift, about 12% better, and slightly bigger than the +0.5pp they'd designed the test to catch. But before celebrating, ask the two questions that matter:

• Is it real, or just luck? This is what a p-value answers: if the button truly made no difference, how likely is a gap this big by pure chance? A small p-value (below 0.05) means luck is an unlikely explanation.
• How big could the true effect be? A confidence interval gives a range, for example "+0.2 to +1.0 points." If that whole range is positive and big enough to matter, you have a real and useful win.

Then check the guardrails (did order value or refunds get worse?) and make the call: ship, don't ship, or iterate.

Plain-words glossary

Mistakes that fool people

• No clear hypothesis — testing random tweaks and fishing for anything that looks good.
• Changing many things at once — change the button and the page, and you won't know which one worked.
• Peeking and stopping early — the single most common way to ship fake winners.
• Watching many metrics — check enough numbers and one will look "significant" by chance. Pick one primary metric up front.
• A broken or lopsided split — if 55% of users land in one group, the test is broken; trust nothing until it's fixed.
• Novelty effect — anything new gets a temporary bump just for being new, and it can fade. Run long enough to see if the lift holds.
• Contamination — the control group somehow sees the treatment (the same person on phone and laptop), which blurs the difference.

When you can't run an A/B test

Some changes can't be split by user — a price change, a country-wide launch, a brand campaign. For these you reason from natural comparisons instead, called quasi-experiments: compare a treated market against similar untreated ones (difference-in-differences), or build a synthetic control from a blend of untreated markets. The mindset is the same — find a fair comparison that shows what would have happened without the change.

What makes a strong answer

In an interview, do design first and statistics second. Name your hypothesis, primary metric, and randomization unit before any math. State the one assumption most likely to break your result before you are asked. And choose the simplest design that is still valid — not the fanciest one.