Interference and SUTVA

By the end of this lesson, you should be able to: recognise the two settings where user-level randomisation breaks, predict which direction the bias runs in each, choose between cluster and switchback designs, and analyse a clustered experiment at the right sample size.

The assumption nobody states

Every A/B test rests on something so obvious it's rarely said out loud: what you do to one user does not affect any other user. Treatment users get the feature, control users don't, and the difference is the effect.

Two things break that assumption, and they don't break it the same way.

One: the effect leaks

Basket tested an invite feature. Treatment users got a button to invite friends; control users didn't.

The true effect on a treated user is +8.00%. Here's what the experiment measured:

The measurement recovers 58% of the truth. The experiment is significant, and it's understated by nearly half.

The reason's in the feature itself. Treatment users invited their friends, and some of those friends were in control. Those control users got an invitation, came back, and ordered. Control's conversion went up, and the gap between the arms closed.

The arithmetic is unforgiving. If a share $\lambda$ of the effect reaches control, you measure:

$\text{measured} = \text{true} \times (1 - \lambda)$

Notice the direction. Leakage always shrinks the measured effect toward zero. That's the reassuring half: a significant result under leakage is still real, just understated, so you aren't going to ship something worthless. The unreassuring half is everything you killed for being flat when it wasn't.