Pinterest — Product Data Science — A/B Test & Causal Inference Questions
Role context: Data Scientist (Product), Pinterest · Est. study time: 55 min · 5 questions
How Pinterest actually experiments
Pinterest is heavily experiment-driven — around 1,000 concurrent experiments on a mature in-house platform — so a product DS reads experiments constantly. This section mirrors how Pinterest's own platform reports them (from its engineering blog), so the questions feel like real experiment reads rather than textbook exercises:
- Assignment is consistent hashing of
hash(experiment_id, user_id)into buckets, so the same user always gets the same treatment. - Significance is an unpaired t-test on metrics like active users and actions (assuming samples are iid, not that the metric is normal — the CLT carries the mean), flagged at p < 0.05.
- Trust starts with a Pearson chi-square group-allocation check — Pinterest's sample-ratio-mismatch guardrail.
- Durability is read from a days-in novelty curve, and effects are sliced across cohorts (country, gender, user state, app type).
- Triggering matters:
activate_experiment()logs only users actually exposed, so you analyze the affected slice, not everyone.
For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.
Each answer is a coaching walkthrough: a Sample answer (clarify the setup → lay out the approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example with the real math, then a Grading rubric.