Segment Effects and Interaction Tests
By the end of this lesson, you should be able to: explain why the average effect can hide very different effects for different users, test whether two segments really differ, avoid the common mistakes in segment analysis, and decide what to do with a real difference.
The average hides the shape
An A/B test reports an average effect: the change raised conversion by 1%, on average across everyone. But "on average" can mean very different things.
- Everyone improved by about 1%.
- Some users improved by 5%, most didn't change.
- Some users improved a lot and others got worse.
These lead to different decisions, and the average alone can't tell them apart. When the effect differs between users, it's called a heterogeneous treatment effect.
A real example: eBay's search ads
eBay tested whether its paid search ads on non-brand keywords were worth the money, by switching them off in some US regions and comparing sales. On average, the ads had a very small effect on sales that wasn't statistically significant. Across all users, they weren't paying for themselves.
Then the researchers split users by how often they had bought on eBay in the year before. They chose this split because of what the economics of advertising predicts: ads should matter most to people who don't already know the site well.
- New users: the ads brought in a significant increase in new registrations.
- Users who had bought only once or twice: the ads significantly increased their purchases.
- Users who had bought three or more times: no significant effect. They knew eBay and came anyway.
The catch: frequent buyers accounted for most of the sales the ads were credited with, so most of the ad spend went on people it didn't influence. That's why the average return was negative, even though the ads worked well for one group.
That's a far more useful finding than "the ads don't work on average". It points to a decision: aim ads at people who don't know eBay yet.