Segment Effects and Interaction Tests

By the end of this lesson, you should be able to: tell a real heterogeneous effect from one manufactured by searching, run the interaction test that actually licenses the claim, decide when a segment finding is worth acting on, and explain why the honest version of this analysis is mostly decided before launch.

A result that says nothing happened

Basket rebuilt its new-user onboarding and tested it on 64,208 users.

Just misses. The team files it under "no effect" and moves on.

Now split by how long the user has been signed up, which is what the onboarding is about:

The onboarding works, strongly, on exactly the people it was designed for. It does nothing for anybody else, which isn't surprising since they'd already been onboarded. The two effects average to something that looks like a null result.

Now the hard part

That was the easy example, because the segment was obvious, named in advance and mechanically sensible. Most segment findings arrive the other way round: somebody scans a breakdown and reports the cell that looks best.

To see what that produces, take Basket's checkout redesign. Its effect is the same for every user by construction, with no heterogeneity anywhere. Slice it 21 ways across country, platform, channel, tenure and past orders:

Australia looks like it responds three times better than average. It doesn't. Every user in that experiment got an identical effect, and the entire 13.3-point spread is sampling noise in smaller cells.

A segment table will always produce a best cell and a worst cell. The spread between them isn't evidence of anything, because a spread that size is what noise looks like when you slice a fixed sample twenty-one ways.