Roblox — Consumer Frontend — A/B Test & Causal Inference Questions

Role context: Senior Data Scientist, Consumer Frontend · Est. study time: 65 min · 6 questions

How experimentation works here

Experimentation on a navigation surface is subtle:

  • The OEC must be successful sessions reached efficiently (qualified play-through, low friction), not clicks or raw time — both rise for the wrong reasons on a navigation surface.
  • A nav/IA change is cross-surface and platform-wide, so a per-user split can leak (interference) or be unavailable — pushing you to geo, switchback, or quasi-experiments.
  • Many frontend metrics are ratios (qPTR) or heavy-tailed continuous (time-to-launch) — analyze them with the delta method and clustered errors, not a naive t-test on the wrong unit.
  • New navigation triggers learning effects (novelty/primacy), so short reads mislead.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (6)

Design an A/B for a Home layout change — what's the OEC?Experiment designHardYour nav change touches every surface — is a user-level A/B valid?InterferenceHardQualified play-through is a ratio metric — analyze it correctlyVariance & ratio metricsHardHow many users (or markets) do you need to detect a 1pp lift?Power & sample sizeMediumThe new nav looks worse on day one — when do you call it?Novelty & durationMediumYour UX dashboard tracks 40 metrics — how do you avoid false wins?Multiple testingHard