Advanced Experimentation
The experiment questions that separate a senior candidate from a competent one. Bayesian readouts, bandits, designs that buy precision, and measuring what a two-week test cannot see.
10 lessons in 3 modules, all running on one product's data and building on each other. Start at the top.
0 of 10 done
Module 1 · Beyond the Fixed-Horizon Test
Reading an experiment as a belief rather than a verdict, and deciding when learning beats measuring.
- 1Bayesian A/B TestingPrior to posterior on a conversion rate, the beta-binomial worked end to end, and P(B beats A): the question VWO's statistician says users ask most. The posterior is honest at every moment, but checking it daily and stopping at 95% ships a winner in about 18% of tests where nothing changed.
- 2Expected Loss and the Decision RuleTurning a posterior into a decision: expected loss, the region of practical equivalence, and why a probability of winning isn't a reason to ship. With VWO's 'threshold of caring' and Microsoft's finding that only about a third of ideas improve the metric they target.
- 3Multi-Armed Bandits and Thompson SamplingThompson sampling, the trade a bandit actually makes, and the two conditions it needs. With Thompson's 1933 idea, Yahoo's evidence that it's 'highly competitive', and a proof that adaptively collected data underestimates the arms it stops sampling.
Module 2 · Designs That Buy Precision
Getting a tighter answer from the same traffic by changing the design rather than the arithmetic.
- 4Stratified and Blocked RandomisationBalancing covariates by construction rather than by luck, plus blocking and re-randomisation. What stratifying can buy is capped by one number, the covariate's correlation with the outcome, and a World Bank study found the gains matter most in small samples.
- 5Cluster Randomisation and the Design EffectRandomising groups instead of users: the intra-cluster correlation, the design effect, and what your effective sample size really is. Why LinkedIn and Facebook built cluster-randomised systems, and why an ICC of just 0.01 can make a user-level analysis fire on nothing 72% of the time.
- 6Switchback ExperimentsTurning the whole system on and off over time when a change touches a shared marketplace. Why Lyft showed a user-level A/B can be six times too big, how DoorDash randomises 30-minute windows by city, and the three traps: time of day, carryover, and the unit of analysis.
- 7Interleaving for Ranking EvaluationBlending two rankings into one list so each user compares them directly. Netflix found it needed more than 100 times fewer members than its most sensitive A/B metric; here's where that gain comes from, why it depends on your data, and what interleaving can't tell you.
Module 3 · The Long Game
Measuring what a two-week test cannot see, and running a platform rather than a test.
- 8Long-Term Holdouts and Effect DecayMeasuring what a two-week test can't see, using a holdout you keep running. With Google's research on ads blindness, which showed short-term results can mislead about the long term, and led Google to cut the ad load on mobile search by half.
- 9Meta-Analysis Across ExperimentsLearning from all your past experiments rather than one: the distribution of true effects, the winner's curse, and shrinkage. Facebook cut the error in 226 News Feed test results by 44% by shrinking them, and Bing's experiments turned out to have very fat tails.
- 10Building an Experimentation PlatformMetric governance, the guardrail checks, and the gates between a readout and a launch. Why careful experiments still produce hundreds of false wins a year at scale, and what Booking.com and Microsoft built to stop them: real-time monitoring, pre-registration, and checks that hide results when the data is broken.