Roblox — Social — A/B Test & Causal Inference Questions

Role context: Data Scientist, Social (Social Connections) · Est. study time: 65 min · 5 questions

How experimentation works for a social graph

Testing social features is subtler than a normal A/B:

  • Effects spread through the friend network, so a user-level A/B violates SUTVA (treated users affect control friends) and is biased — you need cluster / ego-cluster designs.
  • The causal value of a connection is confounded by selection: engaged users make more friends, so the friends-and-retention correlation isn't causal.
  • The right OEC is meaningful connections and co-play, not friend-adds, which a viral feature can inflate.
  • Safety is a hard guardrail on a young platform — and abuse events are rare, so you must power for them and read by age.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

A social feature spreads through friends — how do you A/B it?Experiment designHardDo friends cause retention, or do retained users add friends?Causal inferenceHardA viral invite lifts friend-adds but co-play is flat — ship it?Decide from resultsMediumA social feature spikes then fades — real or novelty?Novelty & durationMediumA growth feature might raise unsafe contact — how do you test it?Metric designHard