Google — YouTube — A/B Test & Causal Inference Questions

Role context: Data Scientist (Product Analytics), YouTube · Est. study time: 55 min · 5 questions

How Google and YouTube experiment

Google runs experimentation at massive scale through its overlapping experiment infrastructure — layers and domains that let many experiments run concurrently with minimal interference, with gradual launch ramp-up (Tang et al., 2010). This section mirrors how YouTube actually reads experiments, so the questions feel like real launch decisions:

  • The hard part is the Overall Evaluation Criterion (OEC) — YouTube optimizes watch time over clicks to avoid clickbait, and pairs it with a satisfaction guardrail because watch time can be gamed.
  • Offline metrics screen; the live A/B decides — "live A/B results are not always correlated with offline experiments" (Covington et al., 2016).
  • Effects are read over a days-in window for novelty and sliced by surface (Home/Search/Shorts) and viewer type.
  • Some metrics (AVD, CTR) are ratios, which need care computing variance.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Watch time is up but satisfaction is down — do you ship?Metric designHardTwo experiments overlap on the same surface — are they interfering?Experiment designHardThe model wins offline but loses online — which do you trust?Decide from resultsMediumThe watch-time lift is fading over the days-in window — real or novelty?Novelty & durationHardAverage view duration is a ratio — why can't you just t-test it?Variance & ratio metricsHard