Causal Inference

Measure an effect when you cannot randomise. Twenty-two lessons built on real studies, with interactive simulations where the true answer is known.

22 lessons in 6 modules, all running on one product's data and building on each other. Start at the top.

0 of 22 done
Start →

Module 1 · What Causal Means

The number you can never see, where the gap comes from, and which variables to condition on.

  1. 1Potential Outcomes and the CounterfactualEvery causal question compares what happened with what would have happened otherwise, and you never see both for the same person. How to write that down, the three different averages people mean by 'the effect', and why randomisation lets you estimate them anyway. With GitHub's randomised experiment on Copilot, where developers with the AI assistant finished a coding task 55.8% faster.
  2. 2Selection BiasComparing people who got a treatment with people who didn't measures the treatment plus how different those people already were. How to split a naive comparison into the real effect and the selection bias, where selection comes from, and how to predict its direction. With Facebook's ads, where selection made the effect look four times bigger, and hormone therapy, where it reversed the sign.
  3. 3Causal Graphs: Confounders, Mediators, CollidersA causal graph draws your assumptions about what causes what, and tells you which variables to control for. The three structures every graph is made of, why controlling for the wrong one adds bias instead of removing it, and why filtering your data is a form of control. With Facebook's ads, Google's ads blindness, and the birth weight paradox.
  4. 4The Back-Door CriterionWhich variables do you need to control for to get a causal effect from observational data? The back-door criterion answers that from the causal graph. How it works, why adding more controls doesn't always help, and why the one confounder that matters can be the one you didn't measure. With Facebook's study, where adding census data barely moved a biased estimate and adding activity during the campaign nearly fixed it.

Module 2 · Adjusting for What You Can See

Regression, propensity scores, matching and weighting, in increasing order of honesty.

  1. 5Regression AdjustmentThe most common way to 'control for' differences between treated and untreated users is a regression with the treatment and some covariates. What that coefficient means, what it rests on, and why adding controls can move the answer away from the truth as easily as toward it. With Facebook's study, where regression gave estimates from 99% down to 37%, and in one campaign the wrong sign, against known experimental answers.
  2. 6Propensity ScoresA propensity score is each user's probability of being treated, given what you know about them. Comparing users with the same score adjusts for all those characteristics at once. How to estimate one, how to use it, how to check overlap and balance, and what it can't fix. With Facebook's study, where matching balanced the variables in the model but left others a full standard deviation apart.
  3. 7Matching and Inverse-Probability WeightingTwo ways to turn a propensity score into a fair comparison: pair each treated user with a similar untreated one, or reweight everyone so the groups look alike. How each works, step by step on a small example, what goes wrong when weights get extreme, and why a perfect balance table doesn't mean a right answer. With Facebook's study, where both methods still overstated the ad effect by close to double.
  4. 8Doubly Robust EstimationA doubly robust estimator combines a model of the outcome with a model of who gets treated, and is right if either one is right. How it works, why it's the default in modern causal toolkits, and what it doesn't protect you from. With Facebook's study, where the doubly robust method did no better than the others, because the problem was a confounder neither model had.

Module 3 · Using Time

Before-and-after and why it fails, then the difference-in-differences family and its assumption.

  1. 9Before-and-After and Interrupted Time SeriesWhen a change ships to everyone, the obvious measure is the metric before against after. Why that usually measures the calendar, how an interrupted time series does better by modelling the trend and the weekly cycle, and what it still can't handle. Worked on real Wikipedia traffic around a 'launch' that never happened.
  2. 10Difference-in-DifferencesWhen one group gets a change and another doesn't, compare how much each one changed. Subtracting the comparison group's change removes everything that hit both groups over the same period. How difference-in-differences works, what it assumes, and how to check it. With the classic study of New Jersey's 1992 minimum wage rise, and Wikipedia traffic showing what a year-earlier comparison removes.
  3. 11The Parallel Trends AssumptionDifference-in-differences only works if the treated and comparison groups would have moved together without the treatment. You can never check that directly. How to look for evidence with an event study, why passing a pre-trend test proves less than it seems, and what to do instead. With the study of Facebook's arrival at US colleges, and Roth's work on how weak pre-trend tests are.
  4. 12Staggered RolloutsProducts often launch in waves: some countries this month, others next quarter. The standard regression for this, two-way fixed effects, can badly mis-measure the effect when it changes over time, and can even get the sign wrong. Why that happens, and the newer estimators built to fix it. With Facebook's college-by-college rollout as a real example.
  5. 13Fixed EffectsA fixed effect gives each unit, or each period, its own baseline, so the comparison happens within a unit rather than across units. That removes every stable difference between units, measured or not. How it works, what it can't remove, and how the study of Facebook at US colleges used it.

Module 4 · One Treated Unit

Building a control group out of other markets, and deciding whether to believe it.

  1. 14Synthetic ControlWhen one city, one country or one state gets a change, no single comparison unit looks enough like it. Synthetic control builds one: a weighted mix of untreated units that tracks the treated unit closely before the change. How it works, why the weights are kept positive and summing to one, and when it fails. With California's 1988 tobacco programme and German reunification.
  2. 15Inference for Synthetic ControlWith one treated unit, the usual standard errors don't apply. How to judge whether a synthetic control gap is real: run the same method on every untreated unit and see where the treated one ranks, move the treatment date back in time, and drop donors one at a time. With the checks behind California's tobacco study and German reunification.
  3. 16Counterfactual ForecastingWhen a change hits everything at once and there's no comparison group, you can still forecast what would have happened from the metric's own history and related series the change didn't touch. How Google's CausalImpact does it, how well it matched a randomised experiment, and how to measure your method's noise floor before trusting it. With real Wikipedia traffic as the backtest.

Module 5 · Manufacturing Randomness

Geo holdouts, instruments, encouragement designs and thresholds: where a design beats an adjustment.

  1. 17Geo Experiments and IncrementalityAd platforms report the sales that followed an ad. What you need is the sales the ad caused. A geo experiment measures that by turning advertising on or off in randomly chosen regions. How Google designs them, how to analyse them, and what they've found: from paid clicks that were 80% incremental to brand ads that were almost entirely wasted.
  2. 18Instrumental VariablesInstrumental variables and two-stage least squares, what makes an instrument valid, whose effect you are estimating, and what a weak instrument costs. With the Vietnam draft lottery, eBay's paid search experiment, and a simulation of 200 studies that share one weak instrument.
  3. 19Encouragement Designs and LATEWhen you can't make anyone use a feature, randomise the invitation instead. Intention to treat, the effect on compliers, why comparing users with non-users fails even inside an experiment, and what a weak nudge costs. With Microsoft's and Accenture's GitHub Copilot experiments and the Coronary Drug Project.
  4. 20Regression DiscontinuityUsing a threshold somebody else already set, choosing a bandwidth, and the assumption that actually gets violated in practice. With Yelp's half-star rounding, which let researchers measure what an extra star does to a restaurant's revenue and its chance of selling out.

Module 6 · Who, and How Sure

Who the effect is biggest for, how wrong it could be, and choosing a method you can defend.

  1. 21Heterogeneous Treatment Effects (CATE)S-, T- and X-learners for estimating who the effect is biggest for, validating a model of something you never observe, and what a targeting rule is worth. With a voter-turnout letter re-analysed by the X-learner's authors, and a causal forest whose varied estimates turned out to be mostly noise.
  2. 22Sensitivity Analysis and Choosing a MethodHow strong an unmeasured confounder would have to be to overturn a result, and a decision procedure for choosing among the methods in this track. With the smoking and lung cancer debate, hormone therapy, a survey of Darfur refugees, and California's tobacco programme, whose parts no method could separate.