Before-and-After and Interrupted Time Series
By the end of this lesson, you should be able to: split a before-and-after number into the parts that have nothing to do with the launch, explain why shipping in response to a bad quarter guarantees a bounce, run an interrupted time series properly, and say exactly what it still cannot rule out.
The number the deck will carry
Alder Stream rebuilt its homepage. Weekly active days for the twelve weeks after against the twelve weeks before:
| Weekly active days | |
|---|---|
| 12 weeks before | 11.085 |
| 12 weeks after | 11.629 |
| Before and after | +4.79% |
The planted truth is +1.19%. The reported number is four times the effect, and there's nothing wrong with the arithmetic.
Taking the 4.79 apart
Because the series was generated, every piece of it can be measured. These four are the only things that differ between the two windows, and they add back exactly.
| Contribution | Share | |
|---|---|---|
| The homepage | +1.19% | 25% |
| The trend it was already on | +2.16% | 45% |
| Where the year sits | +0.57% | 12% |
| Regression to the mean | +0.89% | 18% |
| Sum | +4.81% | |
| Naive gap | +4.79% |
The residual is 1.4 × 10⁻⁴, which is floating-point dust. The homepage is a quarter of the reported lift. The other three quarters would have arrived with no launch at all.
The first two are familiar. Products grow, and twelve weeks of growth lands in the post window and not the pre one. Seasons turn, and two adjacent quarters sit at different points in the year. It's the third that's interesting.