Interference and SUTVA
By the end of this lesson, you should be able to: explain the assumption that interference breaks, recognise the two main ways it happens, describe how it biases a standard test, and choose a design that handles it.
The assumption nobody states
A standard A/B test compares treated users with control users and calls the difference the effect. That only works if each user's outcome depends on their own group and nobody else's. Statisticians call this the stable unit treatment value assumption, or SUTVA.
In many products, it's false. Users message each other, share content, compete for the same drivers, and book from the same set of listings. When treated users change what control users experience, the control group no longer shows what would have happened without the change, and the comparison is biased. This is called interference.
It happens in two main ways.
One: effects spread through connections
In a social product, a change to one user's experience reaches their friends and followers.
LinkedIn tested a new feed recommendation algorithm that showed users a different mix of content. Among the users who got it:
- interactions with feed content fell 12%,
- engagement fell 5%,
- but comments, likes and reshares rose 16%.
A standard test would see a loss: engagement down 5% for treated users. But those extra shares went out to other people. LinkedIn measured this by giving the algorithm to users' connections while the users themselves stayed on the old feed. Users whose connections had the new algorithm engaged 3.1% more (p = 0.03) and scrolled 2.9% more.
So the change hurt the people who received it a little, and helped the people around them. A user-level test misses the second part. It can also hide effects the other way: if control users receive more interesting content from treated friends, control improves too, and the measured difference shrinks.
LinkedIn's fix is called ego-cluster randomisation: assign a user and their direct connections to the same group, so the effect of connections' treatment can be measured. Simpler cluster designs struggle on a social network. LinkedIn notes that splitting its graph into 10,000 clusters still leaves the average member with about 80% of their connections outside their own cluster.