Airbnb — Marketplace — A/B Test & Causal Inference Questions

Role context: Data Scientist, Marketplace Data Science · Est. study time: 60 min · 4 questions

How Airbnb measures the marketplace

Marketplace DS owns the hardest experimentation problem at Airbnb: interference.

  • Buyers and sellers in a market compete for finite supply and demand, so a Bernoulli (user-level) A/B test violates SUTVA and gives a biased total effect — the treated units affect the control units through the market.
  • The fixes are marketplace-specific: cluster randomization by market, choosing listing-side vs guest-side randomization, switchback designs for time-varying changes, and shadow-price / equilibrium corrections. Each trades bias against variance.
  • The decision question is usually created vs shifted — did a change grow the marketplace or just move bookings between listings, markets, or dates.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

Why is a user-level A/B biased in a marketplace, and how do you fix it?Experiment designHardRandomize the guest or the listing? The choice changes your answerTest selectionHardA time-varying marketplace change — why reach for a switchback?Experiment designHardA promo lifts bookings in treated markets — created or just shifted?Decide from resultsMedium