Geo Experiments and Incrementality
By the end of this lesson, you should be able to: explain why attribution and incrementality are different quantities, design a geo holdout, analyse it at the right sample size, and know why the assignment step deserves more care than the analysis.
Two numbers that sound like the same number
Alder Ads reports on the campaign:
| Orders in markets running ads | 1,685,037 |
| Of those, last touch was an ad | 214,000 |
| "Advertising drove" | 12.70% of orders |
Nothing there is wrong. Those 214,000 people really did click an ad, and the ad really was the last thing they touched before buying.
The question a budget needs answered is a different one: how many of those orders would not have happened?
Turn the ads off and find out
Half of Alder's 31 markets went dark from day 84. The other half kept running. Which markets went dark was decided in advance, not by how they were performing.
| Before | After | Change | |
|---|---|---|---|
| Ads stayed on | 3,835 | 4,121 | +7.46% |
| Ads switched off | 3,020 | 3,067 | +1.57% |
Markets that lost advertising entirely still grew 1.57%, because most of the demand was never coming from ads. The gap between the arms is what advertising added:
| Incremental lift | +5.05% |
| 95% interval | [+2.36%, +7.82%] |
| Planted truth | +4.10% |
