Counterfactual Forecasting
By the end of this lesson, you should be able to: forecast a counterfactual and know when you're allowed to, backtest a forecaster before trusting it, report a cumulative effect with a band that widens honestly, and measure what losing the control group cost you.
The case with nothing to compare against
Lessons 14 and 15 measured Bristol's loyalty launch using 30 donor cities. Sometimes those don't exist.
A pricing change ships to every market on the same day. A regulation applies everywhere. A recommendation model is swapped globally. There is no untreated unit anywhere, and every method so far in this track needed one.
What's left is the treated unit's own history. Forecast what it would have done, and subtract.
Bristol is the ideal place to learn this, because we already know the answer. Synthetic control got +5.21% on this exact data, and the planted truth is +5.40%. So the cost of throwing the donors away is measurable rather than hypothetical.
Backtest before you trust it
The whole method rests on a forecast, so the first question is whether the model can forecast this series at all. That's checkable without touching the post-period: fit on an early slice, predict a later pre-treatment slice, and see how far off you land.
| Fit on weeks | Predict weeks | Mean error |
|---|---|---|
| 0–13 | 14–25 | −3.27% |
| 0–17 | 18–25 | −3.09% |
| 0–21 | 22–25 | −3.35% |
Every one of those is negative, and they agree to within 0.14 percentage points. That is not noise, it's bias. The model consistently over-predicts this city by about 3%.
That number deserves respect: a forecaster carrying a 3% systematic error is being asked to measure a 5.4% effect. Most of what you'd claim could be the model's own bias.
The cause is in the data. Alder Go's demand contains a wandering component, and a straight line plus a seasonal term can't represent a series whose trend itself drifts. Extrapolate a fitted slope from a drifting series and you inherit whichever way it happened to be drifting.
If you take one habit from this lesson, take the backtest. It costs nothing, it uses only data you already have, and it's the difference between "the model says +6%" and "the model says +6% and I know it runs 3% hot on this series".