Counterfactual Forecasting

By the end of this lesson, you should be able to: estimate an effect by forecasting what the metric would have done, choose control series that make the forecast trustworthy, read a cumulative effect with its widening interval, and measure the noise floor of your method before believing a result.

When there's nothing to compare against

Sometimes a change reaches everything at once: a national ad campaign, a global launch, a price change for every customer. There's no untreated region, no holdout, no donor pool.

Counterfactual forecasting handles this by predicting what the metric would have done without the change, from two sources:

• The metric's own history: its trend, its weekly and yearly patterns.
• Control series: other time series that move with the metric but that the change couldn't have affected, such as searches for the industry as a whole, or a related product.

The model learns the relationship before the change, forecasts the period after, and the gap between actual and forecast is the estimated effect.

How Google's CausalImpact does it

Google's researchers built a widely used version, released as the CausalImpact package.1 It fits a Bayesian structural time series model, which combines:

1. A trend that can drift over time.
2. Seasonal patterns, such as day of week.
3. A regression on control series, with the model choosing which controls are useful.

Instead of a single forecast, it produces a range of plausible forecasts. That gives an interval for the effect on each day, and for the cumulative effect over the whole period.