Before-and-After and Interrupted Time Series
By the end of this lesson, you should be able to: explain why a simple before-and-after comparison is usually biased, build an interrupted time series that accounts for trend and weekly patterns, read its result, and name the situations where it still fails.
The comparison everyone reaches for
A change ships to every user on the same day. There's no control group. The obvious way to measure it is to compare the metric in the weeks before with the weeks after.
The problem is that the "after" period differs from the "before" period in more ways than the change. Time moves on: seasons, holidays, news, marketing, competitors, and the slow growth or decline of the product. A before-and-after comparison credits all of that to the change.
A launch that never happened
Here's a real test of the method. Take English Wikipedia's daily desktop traffic, and pretend a change launched on Monday 3 November 2025. Nothing actually launched that day. So any "effect" a method finds is an error.
| Method | "Effect" of a launch that didn't happen |
|---|---|
| Four weeks after vs four weeks before | +4.1% |
| Interrupted time series (trend and weekday pattern, 8 weeks before) | +1.7% |
| The same four weeks a year earlier, 2024 | +2.6% |
A before-and-after comparison would have reported a 4.1% lift from a launch that didn't exist. The last row shows why: traffic rose over the same weeks in 2024 too, by 2.6%, with no launch either. Early November is simply busier than October. The calendar did most of the work.