Geo Experiments and Incrementality
By the end of this lesson, you should be able to: explain the difference between attributed and incremental results, design a geo experiment, analyse it to get the return on ad spend, and choose another way to hold out when geos won't work.
Two numbers that sound the same
An ad platform tells you how many clicks, visits or sales followed your ads. That's the attributed number. What you need for a budget decision is the incremental number: how many of those wouldn't have happened without the ads.
The gap can be small or enormous. Google's own paper on geo experiments gives an example from one of its advertisers. The platform reported a cost per click of $2.40. The experiment found each dollar of ad spend produced a third of an extra click, so the cost per incremental click was $3. About 80% of the paid clicks were truly extra. The other 20% would have happened anyway, through the free search results.
At eBay, the gap was much bigger. When eBay switched off search ads on its own brand name, 99.5% of the lost ad clicks came back through free search results (Product Analytics lesson 19). Almost none were incremental.
You can't tell which situation you're in from the platform's report. And you can't fix it with a regression on your own spending data either. Before its experiment, eBay regressed revenue on search ad spend across regions and days.
| Estimate | Revenue gain from 10% more spend |
|---|---|
| Simple regression | 9% |
| Adding region and day controls | 1.3% |
| The geo experiment | Paid search as a whole added only 0.66% to sales |
Spending follows searches, and searches follow people who were already going to buy. That's the confounding from lesson 2, and controls only removed part of it. The experiment found a return on investment of −63%.
Turn the ads off and find out
You usually can't randomise ads person by person: people see ads on many devices and channels, and many outcomes, like in-store sales, can't be linked to an individual. A geo experiment randomises regions instead.
- Divide the market into non-overlapping regions, called geos. In the US, a common choice is the 210 designated market areas (DMAs) that advertising platforms already target.
- Randomly assign each geo to treatment or control.
- Run a pretest period where all geos get the same campaign, to measure how each geo normally behaves.
- Run a test period where the treatment geos get a different campaign: ads switched off, ads switched on, or spend increased.
- Compare the change in the outcome in treatment geos with the change in control geos.
Because the geos were randomised, the difference measures what the ads caused, not just what followed them.