Advanced Experimentation
The experiment questions that separate a senior candidate from a competent one. Bayesian readouts, bandits, designs that buy precision, and measuring what a two-week test cannot see.
10 lessons in 3 modules, all running on one product's data and building on each other. Start at the top.
0 of 10 done
Module 1 · Beyond the Fixed-Horizon Test
Reading an experiment as a belief rather than a verdict, and deciding when learning beats measuring.
- 1Bayesian A/B TestingPrior to posterior on a conversion rate, the beta-binomial worked end to end, and P(B beats A). The posterior is honest at every moment, but monitoring it daily and stopping at 95% fires 19.7% of the time on two identical arms, because the stopping rule breaks the same way it does for a p-value.
- 2Expected Loss and the Decision RuleTurning a posterior into a decision: expected loss, the region of practical equivalence, and why a probability of winning is not a reason to ship. Meridian's rebooking test is 99.4% likely to be better and only 59% likely to clear its own break-even.
- 3Multi-Armed Bandits and Thompson SamplingEpsilon-greedy and Thompson sampling, and the trade they actually make. Thompson beat an equal split by 3.47% on the same 240,000 users, and ended with each losing arm estimated three times more noisily, because it stopped sending them traffic.
Module 2 · Designs That Buy Precision
Getting a tighter answer from the same traffic by changing the design rather than the arithmetic.
- 4Stratified and Blocked RandomisationBalancing covariates by construction rather than by luck, plus blocking and re-randomisation. Across 4,000 re-randomisations, simple assignment was unbiased and widely scattered; stratifying cut that scatter by 56%, and a single experiment needs the scatter, not the average.
- 5Cluster Randomisation and the Design EffectRandomising groups instead of users: the intra-cluster correlation, the design effect, and what your effective sample size really is. An ICC of 0.01 across 3,000 users per city gives a design effect of 31, and a test that ignores it fires on a true null 71.3% of the time.
- 6Switchback ExperimentsTurning the whole system on and off over time when a change touches a shared marketplace. Meridian's surge change truly cuts ETA 6.2%; a user-level A/B measures 3.47% and tightens confidently around the wrong number, while a switchback recovers 7.04%.
- 7Interleaving for Ranking EvaluationBlending two rankings into one list so each user compares them directly. Between-session variation on Meridian's data is 21 times the gap between the rankers, which is the setup interleaving exists for, and it still came out level with an A/B test on clicks.
Module 3 · The Long Game
Measuring what a two-week test cannot see, and running a platform rather than a test.
- 8Long-Term Holdouts and Effect DecayMeasuring what a two-week test cannot see, using a holdout you keep running. The same feature reads +10.2% in week one and +2.5% in week twenty-six, both correct, and nothing in the short readout tells you which one you are looking at.
- 9Meta-Analysis Across ExperimentsLearning from the whole corpus rather than one test: empirical priors, the winner's curse, and a win-rate reality check. Across 220 experiments with known truths, the average significant winner reported +4.60% against a true +2.86%, and shrinking toward the corpus prior cut the error by 43%.
- 10Building an Experimentation PlatformMetric governance, the guardrail catalogue and the gates between a readout and a launch, which is what an interviewer means by 'how would you build this'. 220 correct experiments against 20 metrics each produce hundreds of false significant readings a year by arithmetic alone.