Experimentation

Design an A/B test, size it, and read it without fooling yourself. Twenty lessons on one platform's experiments, where the true answer is known.

20 lessons in 5 modules, all running on one product's data and building on each other. Start at the top.

0 of 20 done
Start →

Module 1 · Foundations

What randomising buys you, and the four decisions to make before launch.

  1. 1Randomisation and What an Experiment ProvesWhat randomisation buys you, and why the two obvious alternatives fail. On a feature whose true effect is +3%, before-and-after says +19% and adopters-versus-non-adopters says +74%. Randomising says +3.45%, and it is the only one of the three that stays right over 200 repeats.
  2. 2Pre-Registration and the Design DocWhat to decide before launch and why it has to be beforehand. One experiment with no effect at all has 363 defensible ways to analyse it, five of which return p below 0.05. The pre-registered analysis says p = 0.559, and nothing else is allowed to decide the launch.
  3. 3Choosing the Randomisation UnitPicking the unit to randomise, and the separate decision of which unit to analyse. Splitting by session lets 82.2% of users see both variants; splitting by user but analysing by session pushes the false positive rate to 12.1%. Both mistakes are invisible in the readout.
  4. 4The OEC and Guardrail MetricsChoosing the one metric a decision turns on, and the guardrails that stop it being gamed. Revenue needs 13.8 weeks to resolve a 2% effect where sessions per user needs 0.7, and the most sensitive candidate is disqualified outright for conditioning on an outcome.

Module 2 · Running an Experiment

Sizing it, proving the split is sound, and checking the instrument.

  1. 5Power and Sample SizeSizing a test, and turning users into days. Detecting a 10% lift on Basket's checkout takes two days; detecting a 1% lift takes seven months. The gap between those two answers is most of experiment design, and most tests are sized by nobody.
  2. 6A/A Tests and Platform ValidationTesting nothing against nothing to check the platform itself. Over 4,000 null experiments a healthy platform declares a winner 5.3% of the time and spreads its p-values evenly; the wrong analysis unit gives 6.7%, and a leaky assignment table 8.7%.
  3. 7Sample Ratio MismatchThe chi-square check on the user counts, and why it invalidates a result outright rather than widening an interval. A shelf that read +8.15% against a true +5.00% was caught by the counts, and three of the six standard balance checks never noticed.
  4. 8Experiment Data QualitySymmetric faults barely matter and asymmetric ones destroy the answer. A logging gap hitting both arms moves the read from +3.51% to +3.86%; the same gap hitting treatment alone turns a real +5.5% into -6.21% with p below 0.0001, and the split check does not notice.
  5. 9Reading a Result: P-Values and Confidence IntervalsWhat the two numbers actually mean, measured rather than defined. A thousand experiments on the same true +5.5% returned estimates from -3.50% to +16.60%; the 95% interval covered the truth 96.3% of the time and the test called it significant in only 39.3%.

Module 3 · Analysing It

Choosing the test, handling awkward metrics, and buying back precision.

  1. 10Choosing the TestMetric shape to test, and why the choice matters less than people think. Five of six tests hold their 5% error rate, including the ones the warnings are about. What breaks a readout is analysing a different unit than you randomised.
  2. 11Ratio Metrics and the Delta MethodStandard errors when the denominator is random too. On a CTR of 97,658 clicks over 1,156,407 impressions the naive standard error is 0.93 times the truth, pushing the A/A false positive rate to 6.3%. The delta method matches the bootstrap to three decimals.
  3. 12Heavy Tails and CappingWhat a long tail costs a standard error, and what capping buys and costs. Basket's top 1% of users hold 17.4% of revenue and 39.9% of the variance; capping at the 99th percentile cuts the standard error 11% and stops measuring 4.6% of the money.
  4. 13CUPED and Variance ReductionBuying precision with a column you already have. Pre-period spend correlates with the test metric at 0.32, and spending that correlation cuts variance by exactly 10.5% without touching the estimate. The derivation, the traps, and whether 10% is worth it.

Module 4 · The Traps

Five ways a correct calculation still gives you the wrong answer.

  1. 14Sequential Testing and PeekingWhat checking every day costs, and the corrections that make it valid. A null experiment watched daily for three weeks crosses 5% significance 19% of the time, because the error rate you designed for only applies if you look once.
  2. 15Multiple Testing: FWER and FDRBonferroni, Benjamini-Hochberg, and which error rate you actually want to control. Twenty secondary metrics generated with no effect at all produced exactly one winner at p = 0.032, which is precisely what twenty tests at 5% predicts.
  3. 16Novelty and Primacy EffectsTelling a fading effect from a real one, and why running longer does not fix it. A banner lifted daily orders 13.15% on day one and 0.55% by week three, and every stopping point along the way says ship. The steady-state effect is nothing.
  4. 17Interference and SUTVAWhen your control group is not untreated: effect leakage and shared resources. An invite feature with a true +8.00% measures +4.68% because treatment users invite control users, and on a courier network speeding up one order slows another.
  5. 18Triggered Analysis and DilutionAnalysing the users who actually saw the feature, and the much worse mistake sitting next to it. A feature reaching 12.2% of assigned users reads +2.41% on everyone and +24.25% on those who triggered, and the gap is exactly the trigger rate.

Module 5 · Deciding

Who the effect is for, and turning a readout into a recommendation.

  1. 19Segment Effects and Interaction TestsThe test that licenses a claim about a segment, and why eyeballing segments does not. An onboarding change reads +3.12% overall and +8.01% for first-month users, but a segment search on a perfectly uniform experiment produces an apparent 13.3% spread.
  2. 20Shipping the DecisionOne experiment read start to finish, applying every check in the track. The split is clean, the primary metric reads +1.54% with p = 0.30, support contacts are up 23.65%, and the feature genuinely works. The correct decision is still not to ship.