Inference for Synthetic Control

By the end of this lesson, you should be able to: produce a p-value for a synthetic control without assuming anything, use the statistic that removes a judgement call from that test, and run the two robustness checks that can actually fail.

The problem with lesson 14's placebo test

Lesson 14 ended with a placebo test: pretend each donor city was treated, refit, and see whether Bristol's gap stands out. Seven donors made it into that test. Twenty-three were dropped for having a pre-period fit "more than 2.5 times worse than Bristol's".

That filter's doing something reasonable and something indefensible at the same time.

Reasonable: a city that no blend of its peers can reproduce genuinely tells you nothing. Its post-period gap is large because the method never fitted it, not because anything happened.

Indefensible: the threshold was chosen by me, and moving it changes the p-value. Any test where the analyst picks which comparisons count is not a test.

One statistic, no filter

The fix is to stop excluding badly-fitted units and start dividing by how badly they fit.1

$\text{ratio}_i = \frac{\text{RMSE}_i^{\text{post}}}{\text{RMSE}_i^{\text{pre}}}$

A city the method reproduces well pre-treatment has a small denominator, so a real post-period gap makes the ratio large. A city it never reproduced has a large denominator, so its large post-period gap divides down to nothing.

The filter becomes part of the statistic. Nobody chooses a threshold.

For Bristol:

Now run the identical procedure on every donor. Twenty-nine of thirty converged, and none was excluded for fit quality:

Bristol ranks first of thirty, and it isn't close: 18.82 against a next-best of 3.95.

$p = \frac{1}{30} = 0.033$

Notice that 0.033 is also the smallest p-value this design can produce. With 30 units, the treated one can at best rank first out of thirty. If someone demands p below 0.01 you need more donors, and that's a decision to make before running, not after.