DoorDash — Merchant, Ads & Sales — A/B Test & Causal Inference Questions

Role context: Data Scientist, Merchant/Ads & Sales Analytics · Est. study time: 50 min · 4 questions

How experimentation works here

Ads experimentation is causal and interference-laden:

  • Ad value is incremental, not attributed — measure the orders an ad caused versus a holdout, separating cannibalized organic.
  • Ads are an auction, so a per-merchant A/B is biased — merchants compete for the same slots and consumer attention (one's win is another's loss), and budgets couple them.
  • The OEC balances ad revenue, consumer experience, and merchant value — never ad revenue alone.
  • Ad ROAS is heterogeneous — high for small/new merchants, low for already-popular ones.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

Design an A/B test for an ads ranking change — what's the OEC?Experiment designMediumMeasure whether ads create incremental orders (not cannibalize organic)Causal inferenceHardWhy a per-merchant ads A/B is biased — the auction problemInterferenceHardWhich merchants should get ads? Heterogeneous ROASHeterogeneous effectsHard