Incrementality Testing (Ghost Ads, PSA & Geo Lift)
By the end of this article, you should be able to: explain why comparing people who saw an ad with people who didn't over-credits the ad, design a holdout that measures the real effect, fall back to a geo experiment when you can't hold out people, and size a test that can actually detect the effect.
The sales that would have happened anyway
An ad's incremental effect is the sales it caused: the ones that wouldn't have happened without it. That's different from the sales it touched. Someone who searches "ebay shoes", clicks eBay's paid ad and buys a pair touched the ad. If they'd have clicked the free link right below it and bought anyway, the ad caused nothing.
eBay tested this directly. In 2012 it stopped paying for brand-keyword ads (searches containing "eBay") on Yahoo and Microsoft's search engines, while keeping them on Google for comparison. Almost all the traffic and sales the paid ads had been credited with arrived through the free search results instead. The researchers found "no measurable short-term value in brand keyword advertising."
Then eBay ran a bigger test on its other search ads, the ones for terms like "used gibson les paul". It stopped bidding on all non-brand keywords for 60 days in a random 30% of US regions and compared sales with the other 70%. The difference was stark:
| Method | Estimated return on paid search |
|---|---|
| Regression on observational data, no controls | over +4,100% |
| Regression with time and region controls | over +1,400% |
| Geo experiment | −63% (95% interval −124% to −3%) |
Read from the top: the standard analysis said paid search was hugely profitable. The experiment said it lost money. The ads did help new and infrequent buyers. But most of the spending went on frequent eBay shoppers, who would have bought anyway.
Why exposed-versus-unexposed misleads
The usual dashboard compares the purchase rate of people who saw the ad with people who didn't. That comparison is broken in a specific way: the ad platform decides who sees the ad, and it shows ads to people it predicts will buy. So the exposed group would have bought more even without the ad. This is selection bias, and it inflates the measured effect.
The explorer gives the ad a true lift of +5%. With no targeting at all (slide the first control to zero), the exposed-versus-unexposed comparison gets it right: +5.0%. Turn targeting up to 1.5 and the same comparison (the line marked Naive) reports +186%. The ad didn't get better; the platform just showed it to people who were going to buy.
Now set the true lift to zero. The comparison still reports +173%. A completely useless ad looks like a huge success.
This isn't only a simulation. Facebook ran 15 large advertising experiments in 2015, with 500 million user-experiment observations, and then estimated the same effects with the observational methods advertisers commonly use, including careful matching. The observational methods generally overestimated the effect, and in half of the studies the estimated increase in purchases was off by a factor of three.