Causal Impact (Counterfactual Forecasting)

By the end of this article, you should be able to: build a forecast of what a metric would have done without a change, read the gap between actual and forecast as the change's effect, tell a real gap from noise, and run the checks that make the estimate credible.

When there's no control group

Some changes reach everyone at once. A new notification algorithm ships to every user on the same day. A marketing campaign runs nationwide. A pricing change applies to all customers. There's no untreated group to compare with, so an A/B test is off the table.

A before-and-after comparison is the obvious fallback, and it's misleading. It credits the change with everything else that moved the metric at the same time: the usual trend, the season, a competitor's launch, a holiday.

What you need is the counterfactual: what the metric would have done if the change had never happened. When no group can supply it, you can forecast it.

Forecast the missing half

The idea fits in three steps:

1. Learn how the metric behaved before the change: its trend, its weekly pattern, and its relationship to other series the change didn't touch.
2. Project that forward over the period after the change, as if nothing had happened. That projection is the counterfactual.
3. Read the gap. The effect on day $t$ is

$\tau(t) = Y_{\text{actual}}(t) - \hat{Y}_{\text{forecast}}(t)$

and the total effect is the sum of those gaps over the period after the change.

The forecast is much better when it can lean on control series: other time series that move with your metric but weren't affected by the change, such as the same metric in a region that didn't get it, or industry-wide search interest. They carry the season and the shocks the change didn't cause, so the gap is left with only what the change did. This is the time-series cousin of synthetic control.