Meta — Instagram — A/B Test & Causal Inference Questions

Role context: Data Scientist, Product Analytics (Instagram) · Est. study time: 55 min · 4 questions

How Instagram experiments

Instagram is a two-sided viewer/creator graph, which shapes how it tests:

  • A feed-ranking change randomized on viewers moves creators through spillover, so a viewer-level test can't cleanly read a creator-side metric — you need cluster / ego-cluster designs.
  • The Overall Evaluation Criterion is subtle: watch time can be gamed, so it travels with sends (a costlier signal) and integrity guardrails, and the signals can disagree.
  • New formats spike then decay, so effects are read over a days-in window.
  • Small effects on huge surfaces require care about power and detectability — the exact thing the Analytical Execution round probes.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

You randomized viewers but need a creator metric — why can't you read it?Experiment designHardWatch time is up but sends are down — is it a real win?Decide from resultsHardA new Reels format spikes then fades — real or novelty?Novelty & durationMediumHow long must you run a test to catch a 1% watch-time lift?Power & sample sizeHard