A Feed change spreads between friends — is your A/B test lying? — Meta Facebook Data Scientist Interview

A Feed change spreads between friends — is your A/B test lying?

Experiment designHard

Problem. You're testing a Feed change designed to increase resharing. You randomize users 50/50 and measure reshares. A colleague says the result will be biased. Why might they be right, and how would you design the test properly?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Does the feature's effect travel between people — if a treated user reshares more, do their friends (some in control) see and reshare more too?"
"What's the estimand — the effect if we launched to everyone, or the effect on an isolated individual?"
"Roughly how connected is the affected population — dense friend clusters or independent users?"

Lay out your approach

Name the violation. Standard A/B assumes SUTVA: each user's outcome depends only on their own treatment. Resharing breaks it — a treated user's extra reshares land in control friends' feeds, lifting control's reshares too. Control gets partially treated, the measured gap shrinks, and we underestimate the true launch effect.
Fix it by changing the randomization unit. Randomize clusters of connected users (graph cluster randomization) instead of individuals, so most of a person's friends share their assignment and spillover stays within a cluster rather than leaking across arms.
Estimate correctly. Define "network exposure" and use an inverse-probability (Horvitz-Thompson) estimator to stay unbiased under interference.
Accept the cost: clusters are correlated, so effective sample size drops and you need more units/power (often several times more).

Then check in: "The bias is interference — reshares spill from treated users into control friends, shrinking the gap. I'd switch to cluster randomization and an inverse-probability estimator. Want me to work through the bias direction and the estimator?"

Simulate the back-and-forth

Interviewer

Which direction is the bias, and why does it matter?

You

It biases the effect toward zero for a viral feature. Because treated users' reshares reach control friends, control's reshare rate rises above its true no-treatment level, so treatment-minus-control understates the real effect of launching to everyone. That matters because we might kill a genuinely good viral feature for looking weak, or under-credit its impact. For a feature that suppresses sharing, the same leakage would bias the other way — the point is the estimate is wrong, and predictably so.

Interviewer

How does clustering actually fix it?

You

If I randomize whole clusters of densely-connected friends, then when a treated user reshares, the friends who see it are mostly in the same (treated) cluster, not in control. Spillover is contained inside the arm rather than crossing between arms, so the between-arm comparison recovers something close to the full launch effect. I'd build clusters with a graph-partitioning algorithm that minimizes edges cut between clusters, since cut edges are the leakage paths.

Interviewer

What do you give up by clustering?

You

Power. Users within a cluster are correlated, so a cluster of 100 friends carries far less independent information than 100 random users — the effective sample size collapses toward the number of clusters, not the number of users. So cluster-randomized tests need materially more units for the same power, often several-fold. There's also a design tension: bigger clusters contain spillover better but cost more power, so I'd size clusters to balance containment against detectability. I'd also consider running both a user-level and a cluster-level test and comparing them — if they disagree, that gap is itself a measurement of the network effect.

Land a clear conclusion

You

So my colleague is right: a user-level test of a viral feature is biased toward zero by interference. I'd randomize clusters of connected users, contain spillover within clusters, and use an inverse-probability estimator for an unbiased launch effect, accepting the extra power cost. If I want to quantify the network effect itself, I'd run cluster-level and user-level designs side by side and read the difference. The one-line summary: when the treatment travels through the graph, move the randomization to the graph.

Key summary — what really matters.

Interference violates SUTVA — treated users' reshares leak into control friends.
Bias is toward zero for a viral feature — you'd understate the launch effect.
Graph cluster randomization contains spillover within clusters.
Inverse-probability (Horvitz-Thompson) estimator stays unbiased under interference.
Cost is power — correlated clusters shrink effective sample size; size clusters to trade off.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you define the launch effect and spot interference.	You treat it as a clean per-user A/B.	You name SUTVA violation and the full-launch estimand.
Design validity	Whether you fix the unit of randomization.	You keep user-level randomization.	You cluster connected users to contain spillover.
Statistical rigor	Whether you know bias direction and the estimator.	"It's biased somehow."	Bias toward zero; inverse-probability estimator; design effect.
Decision & trade-offs	Whether you weigh the power cost.	You ignore the effective-sample-size hit.	You size clusters to trade containment vs power.
Communication	Whether you explain interference plainly.	Jargon with no intuition.	Clear "reshares leak into control, so the gap shrinks."

Interview tips

Strong-answer signal: naming the SUTVA violation, getting the bias direction (toward zero for viral features), switching to cluster randomization with an inverse-probability estimator, and pricing the power cost. Common trap: knowing "network effects are bad for A/B tests" but not the bias direction or the concrete fix.