Netflix — Identity & Commerce — A/B Test & Causal Inference Questions

Role context: Data Scientist, Identity DSE (Commerce) · Est. study time: 70 min · 7 questions

Experimentation in this domain

This section focuses on the experiment-design and causal-inference questions this role is tested on — choosing the unit, sizing and reading a test, variance reduction, sequential monitoring, and the high-stakes decisions you can't cleanly A/B test. (For the probability and statistics fundamentals — p-values, error types, distributions — see the Probability & Statistics section.) Four structural facts make Netflix Commerce experiments distinctive, and the questions lean here:

• The unit of randomization is contested. Account, profile, household, and individual are different units; shared accounts and devices make household interference the default, so SUTVA is the assumption most likely to break.
• The decisions that matter most can't be user-randomized. Price changes and country rollouts fall back to quasi-experiments (synthetic control, DiD).
• The outcomes that matter arrive late. Retention and LTV are right-censored, so you read tests on validated surrogate metrics and reduce variance with CUPED.
• Stakeholders watch continuously. Naive peeking inflates false positives, so use sequential testing / always-valid confidence sequences.

Each answer is a coaching walkthrough: a Sample answer (clarify the setup → lay out the approach → a simulated back-and-forth with the interviewer → a clear call), then a Deep dive with illustrative example carrying the real math and charts, then a Grading rubric.