Difference-in-Differences

By the end of this lesson, you should be able to: compute a difference-in-differences estimate three ways and know why they agree, say precisely what the control group is standing in for, and recognise that the hardest part is not the estimate but the standard error.

A price rise that looks like a win

Alder Stream raised the subscription price in Portugal. Here's what happened to weekly subscribers.

Subscribers went up. A reasonable person reports that the price rise didn't hurt, and might even have helped by signalling quality.

The price rise cost Alder 2.80% of its subscribers. We know because this is generated data and the effect was planted before anything was measured. The before/after read is wrong by 5.25 percentage points and it is wrong in the direction that gets a bad decision approved.

What the before/after is really measuring

Nothing about Portugal was special. Alder Stream was growing in all 24 countries, at roughly the same rate, because it is the same product with the same marketing in different markets.

Over the 13 weeks between the middle of the pre period and the middle of the post period, that growth was worth about 5.4%. A price rise that costs 2.8% sitting on top of growth worth 5.4% nets out to a small increase, which is exactly what the before/after saw.

The before/after credits the price rise with everything else that happened in those weeks. It can't do otherwise, because it has nothing to compare against.