Meta — WhatsApp — A/B Test & Causal Inference Questions

Role context: Data Scientist, Product Analytics (WhatsApp) · Est. study time: 55 min · 4 questions

How WhatsApp experiments

WhatsApp's constraints make experimentation distinctive:

  • End-to-end encryption means you experiment on metadata and proxy metrics (sends, active conversations, retention, blocks), never content — so metric choice and validity carry extra weight.
  • Messaging is networked: a feature given to one person changes their conversation partners' experience, so user-level randomization suffers interference, and geo / cluster designs are common.
  • The app is sticky and near-saturated, so top-line metrics barely move — power and detectability dominate the design.
  • Growth features touch the private core, so rare-event guardrails (spam, blocks) are first-class.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

You can't read messages — how do you pick a metric to experiment on?Metric designHardA messaging feature spreads person-to-person — how do you randomize?Experiment designHardRetention is already 92% — can your test even detect a change?Power & sample sizeHardA growth feature might raise the block rate — which test, and can you see it?Test selectionMedium