Causal Impact — Counterfactual Forecasting

By the end of this article, you should be able to: build a counterfactual forecast for a change you couldn't A/B test, read the gap as the causal effect, confirm whether that gap is statistically real (not just noise), and name the checks that make the estimate credible.

Step 1: Meet the problem

A messaging app rolls out a new notification algorithm to every single user on the same day. There's no control group, no untreated city or market to compare against, because the change went out everywhere at once. A week later, daily messages sent are up. Did the algorithm cause that?

A plain before/after comparison can't tell you. It credits the entire change to your algorithm, including the ordinary week-to-week drift, any seasonal pattern, and anything else happening at the same time. You need to know what would have happened without the change, and there's no other market left to compare against.

Step 2: Forecast the missing half

If there's no untreated group to borrow from, build the missing comparison as a forecast instead. Use the period before the change to learn the metric's trend and its weekly pattern, then project that forward as if the change had never happened. That projection is your counterfactual. The gap between what actually happened and that forecast is the effect:

$\tau(t) = Y_{\text{actual}}(t) - \hat{Y}_{\text{counterfactual}}(t)$

If you do have other signals the change didn't touch, other markets, a correlated but untouched metric, feed them into the same forecast to make it sharper. This is the time-series sibling of synthetic control: same goal, a believable stand-in for "no change happened," built from a forecast here instead of a blend of other units.