Inference for Synthetic Control
By the end of this lesson, you should be able to: test whether a synthetic control gap is bigger than chance using placebo units, check a result by moving the treatment date and dropping donors, and report a synthetic control analysis so others can judge it.
Why the usual standard errors don't work
A synthetic control analysis has one treated unit. The question isn't "how precisely did we measure millions of users?" It's "could a gap this big appear by chance, for any unit, without a treatment?" Standard errors built from the number of users inside the treated unit don't answer that.
The answer comes from running the same method on units where nothing happened, and seeing whether the treated unit stands out.
Placebo units: does the treated unit stand out?
The idea is simple. Pretend each donor was treated instead. Build a synthetic control for it from the other donors, and measure its "effect". Since none of these donors got the treatment, their gaps show how big an effect the method produces by chance.
For California's Proposition 99, Abadie, Diamond and Hainmueller ran this for all 38 donor states. Two adjustments matter.
Some placebos fit badly before the treatment date. A state with a poor pre-period fit will show a big post-period gap too, but that says nothing about chance effects. The authors showed the comparison excluding states whose pre-period fit was more than 20, 5 and 2 times worse than California's. With the strictest cut, 19 states remained, and California's gap still stood out.
A cleaner single statistic is the ratio of the post-period gap to the pre-period gap, using the mean squared prediction error (MSPE) in each period. A large ratio means the unit fit well before and diverged sharply after. California had the highest ratio of all 39 states. If the treatment had been assigned to a state at random, the chance of getting a ratio as large as California's would be 1 in 39, or about 0.026.
That's a permutation p-value: your unit's rank among all units, divided by the number of units. Note what this implies for small donor pools. With 19 donors, the smallest p-value possible is 1 in 20, or 0.05. If you need stronger evidence, you need more donors.