Pinterest — Marketing Data Science — A/B Test & Causal Inference Questions

Role context: Data Scientist, Marketing · Est. study time: 60 min · 5 questions

How marketing measurement works here

Marketing ROI is a causal-measurement problem, and the toolkit is three methods triangulated:

  • Incrementality experiments (geo-lift / holdouts) are the causal ground truth — they measure the lift marketing actually caused, controlling for seasonality and competition by design.
  • MMM (marketing mix modeling) allocates budget across channels using aggregate spend-vs-outcome regression with diminishing-returns curves; powerful but prone to collinearity and spurious fit.
  • MTA (multi-touch attribution) is granular but biased — it credits correlation, not causation, and last-touch over-credits the closing channel.
  • The recurring trap: attribution ≠ incrementality. A channel can be credited with conversions it didn't cause. Experiments calibrate the models.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Design a geo-holdout to measure a channel's true incremental revenueExperiment designHardMMM, attribution, or incrementality — which do you use when?Test selectionHardLast-touch says paid search wins — why don't you believe it?Causal inferenceMediumYour geo test shows +5% but you only have 20 markets — is it real?Decide from resultsHardYour MMM says email ROI is huge — why might the model be lying?Causal inferenceHard