Meta — Facebook — A/B Test & Causal Inference Questions

Role context: Data Scientist, Product Analytics (Facebook) · Est. study time: 55 min · 4 questions

How Meta experiments

Meta runs experiments constantly, but Facebook is a social graph, so the textbook A/B test needs care:

  • A change to one person's Feed spills over to their friends, so treatment leaks into control. That violates SUTVA and biases naive user-level tests — the reason Meta pioneered graph cluster randomization (cluster connected users, randomize clusters, estimate with an inverse-probability estimator).
  • The hard part of any launch is the Overall Evaluation Criterion: an engagement lift with an integrity regression is not a win, so reads pair MSI-style engagement with integrity guardrails.
  • Effects are read over a days-in window for novelty and across a large scorecard, which raises multiple-testing false positives.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

A Feed change spreads between friends — is your A/B test lying?Experiment designHardMSI is up but integrity is down — do you ship?Decide from resultsHardA Groups notification spikes sessions week one — real or novelty?Novelty & durationMediumYour Feed scorecard has 60 metrics — how many 'wins' are fake?Multiple testingHard