Snap — Core — A/B Test & Causal Inference Questions

Role context: Senior Data Scientist, Core · Est. study time: 65 min · 6 questions

How experimentation works here

Experimentation on a communication app is subtle:

  • The OEC must be reciprocated communication + retention, not raw sends or time (both are gameable and one-sided).
  • Snapchat is a friend network, so a communication feature spreads through the graph — interference breaks a naive user-level A/B.
  • Many features trigger for only a subset, so whole-population metrics dilute the effect.
  • Snap proves app performance (latency/crashes) causally moves retention — and measuring that has real pitfalls.
  • Metrics are a governed asset (Snap's STOMP): check the split (SRM) and control the error rate before believing a result.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (6)

Design an A/B for a messaging feature — what's the OEC?Experiment designHardYour feature spreads through the friend graph — is the A/B biased?InterferenceHardOnly 20% of users are affected — why does the metric look flat?Experiment designMediumDoes the app being slower actually cause churn?Causal inferenceHardSends are up week one — but will it retain?Novelty & durationMediumIs your experiment's split even trustworthy?Multiple testingHard