Reddit — Consumer — A/B Test & Causal Inference Questions

Role context: Senior Data Scientist, Consumer · Est. study time: 60 min · 5 questions

How experimentation works here

Experimentation on a community platform is subtle:

  • The OEC must be meaningful engagement + retention + community health, not raw time (engagement is gameable by outrage), and must respect the create→consume loop.
  • Content and votes are shared, so a feed change affects what everyone sees — interference and feedback loops.
  • The JD wants experiment meta-analysis — learning across many A/Bs (winner's curse, multiple testing, heterogeneity).
  • Short-term engagement can diverge from long-term retention and community health.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Design an A/B test for a feed change — what's the OEC?Experiment designHardA change lifts consumption but cuts contributions — ship it?Metric designHardYour feed change spreads through communities — is the A/B biased?InterferenceHardLearn across 200 experiments without fooling yourselfDecide from resultsHardEngagement is up week one — but will it retain?Novelty & durationMedium