Twitch — Monetization — A/B Test & Causal Inference Questions

Role context: Data Scientist, Monetization · Est. study time: 65 min · 5 questions

How experimentation and causal inference work here

Monetization at Twitch is unusually causal-inference heavy:

  • Consumer features (Gifting UX, sub prompts, Bits packages) get high-velocity A/B tests — but the OEC must be revenue with retention and experience guardrails.
  • You usually can't A/B a price (legal/ethical/backlash, and subs are sticky), so pricing effects come from quasi-experiments — difference-in-differences, synthetic control, elasticity from natural variation.
  • Effects are heterogeneous (whales vs casual, region, tenure), so the role uses double ML and causal forests to estimate conditional effects and target.
  • Gifting has network effects, so a naive user-level A/B is biased by spillover — cluster designs and incremental-conversion reads are needed.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Design an A/B test for a new Gifting prompt — what's the OEC?Experiment designHardYou can't A/B a price change — how do you measure its effect?Causal inferenceHardWho should get the discount? Estimating heterogeneous effectsHeterogeneous effectsHardA Gifting experiment looks great — but is it just network spillover?InterferenceHardA promo lifts sub conversion 20% — do you ship it?Decide from resultsMedium