Experimentation
Design an A/B test, size it, and read it without fooling yourself. Twenty lessons built on experiments that real companies have published.
20 lessons in 5 modules, all running on one product's data and building on each other. Start at the top.
0 of 20 done
Module 1 · Foundations
What randomising buys you, and the four decisions to make before launch.
- 1Randomisation and What an Experiment ProvesWhy companies like Microsoft test ideas instead of trusting judgement, why before-and-after comparisons and comparing users who chose a feature both mislead, and what random assignment fixes. With Facebook's ad experiments, where comparing exposed users said +316% and the randomised test said +73%.
- 2Pre-Registration and the Design DocWhy the analysis has to be decided before the data arrives, what a design document contains, and how Microsoft's experimentation team checks a test before it starts. With the study where 29 teams analysed the same data and got answers ranging from slightly fewer red cards to nearly three times as many.
- 3Choosing the Randomisation UnitUsers, devices, sessions, teams or regions: what gets randomised changes what you can measure, how much traffic you need, and whether the comparison is clean. How assignment actually works, and why analysing a smaller unit than you randomised makes false positives far more common.
- 4The OEC and Guardrail MetricsAn experiment needs one metric that decides success and a set of metrics that must not get worse. How to choose them, with the Bing bug that made search results worse and raised revenue per user by 30%, and how Airbnb's guardrail system decides which experiments need a second look.
Module 2 · Running an Experiment
Sizing it, proving the split is sound, and checking the instrument.
- 5Power and Sample SizeHow many users an experiment needs, and why. The four quantities that set sample size, why halving the effect you want to detect quadruples the users you need, and what goes wrong when a test is too small. With Bing's slowdown experiment, where every 100 milliseconds cost 0.6% of revenue.
- 6A/A Tests and Platform ValidationAn A/A test gives both groups the same product, so any 'effect' it finds is a bug or chance. How to use A/A tests to check an experimentation platform, why the whole spread of p-values matters more than one result, and what Microsoft's A/A tests have caught: bots, a bucket assignment bug, and effects that carried over for three months.
- 7Sample Ratio MismatchWhen the groups in an experiment aren't the sizes you configured, the result can't be trusted, and it can even point the wrong way. How to check for a sample ratio mismatch, why about 6% of Microsoft's experiments have one, and where they come from. With the MSN test that looked like a loss until bot filtering was found to be removing the treatment's most engaged users.
- 8Experiment Data QualityA logging problem that hits both groups equally costs you some precision. One that hits one group more than the other can invent a result. How to tell the two apart, why the change being tested is so often what breaks the logging, and the checks to run before reading any metric. With Bing's experiments where a slower page looked like it got more clicks.
- 9Reading a Result: P-Values and Confidence IntervalsWhat a confidence interval and a p-value actually say, what they don't, and why a significant result is wrong more often than 5% of the time. With the American Statistical Association's principles on p-values, and Kohavi, Deng and Vermeer's estimates showing that at companies where about 10% of ideas work, about 22% of significant wins are false.
Module 3 · Analysing It
Choosing the test, handling awkward metrics, and buying back precision.
- 10Choosing the TestWhich test to use for a conversion rate, an average, a ratio or a skewed revenue metric, and which common warnings you can ignore. Why the t-test works on data that isn't normal, why '30 users is enough' is far too few for revenue, and why two metrics both called 'click-through rate' can give different answers.
- 11Ratio Metrics and the Delta MethodClick-through rate per page view, revenue per order and queries per session are ratios whose denominator isn't the randomised user, and the obvious standard error for them is wrong. How the delta method fixes it using only per-user totals, and what happens to the false positive rate if you skip it.
- 12Heavy Tails and CappingWhen a few users account for most of a metric, its variance explodes and experiments need huge samples. How capping fixes that, what it changes about the metric, and how to choose a cap honestly. With Bing's revenue per user, where a $10-a-week cap cut skewness from 18 to 5.3 and let the same test detect changes 30% smaller.
- 13CUPED and Variance ReductionMuch of the noise in an experiment comes from users simply being different from each other, and you already know how they behaved before the test. CUPED uses that to cut the noise. How it works, why it doesn't bias the result, and what it bought Bing: about half the variance, the same as doubling traffic.
Module 4 · The Traps
Five ways a correct calculation still gives you the wrong answer.
- 14Sequential Testing and PeekingChecking an experiment every day and stopping as soon as it looks significant makes false positives several times more common. Why that happens, how sequential methods let you look early without the cost, and why stopping a harmful test early is fine. With the always-valid p-values developed at Optimizely, and Microsoft's alerting.
- 15Multiple Testing: FWER and FDRAn experiment that tracks 50 metrics will almost always show a 'significant' change in one of them, even when nothing happened. Why, what you can control (the chance of any false win, or the share of wins that are false), and how to set up metrics and segments so the readout means something. With Airbnb's calculation that 50 guardrails at 5% give a 92% chance of a false alert.
- 16Novelty and Primacy EffectsAn effect measured in the first weeks of an experiment isn't always the effect you'll get after launch. Users try new things out of curiosity, struggle with changes before adapting, and slowly learn to ignore what doesn't help them. How to tell these apart from noise, with MSN's Hotmail link, Microsoft's 'trends' that came from an A/A test, and Google's study of ads blindness.
- 17Interference and SUTVAA standard A/B test assumes each user's outcome depends only on their own group. In social products and marketplaces that's often false: effects spread between friends, and groups compete for the same drivers or listings. How interference biases results, and how LinkedIn, DoorDash and Airbnb design experiments around it.
- 18Triggered Analysis and DilutionWhen a change only affects users who reach a certain screen or situation, analysing everyone dilutes the effect and wastes traffic. How to analyse only the users who could be affected, why that needs the same trigger logged in control, and how to report the result. With Microsoft's guidance on counterfactual logging and Bing's malware-ad experiment on 3.8 million triggered users.
Module 5 · Deciding
Who the effect is for, and turning a readout into a recommendation.
- 19Segment Effects and Interaction TestsAn average effect can hide a change that helps some users and does nothing, or harm, for others. How to look for differences between segments without fooling yourself, why 'significant in one segment but not the other' isn't evidence of a difference, and when a segment result is worth acting on. With eBay's search ads, which worked for new and occasional buyers and not at all for the frequent ones who drew most of the spend.
- 20Shipping the DecisionEverything in this track as one procedure for deciding whether to ship: check the experiment is valid, read the primary metric against what matters, ask what the test could have detected, check the guardrails, ask whether the result is believable, and look for who it helps and how it lasts. Each step illustrated with a real decision at Bing, MSN or elsewhere.