Airbnb — Guest & Host Product — A/B Test & Causal Inference Questions

Role context: Data Scientist, Guest & Host Product DS · Est. study time: 55 min · 4 questions

How Airbnb experiments on the guest & host product

Discovery and trust drive the marketplace, and both make experimentation subtle:

  • Ranking experiments must handle position bias (higher-ranked listings get clicked because they're higher) and the fact that offline ranking gains don't always hold online — Airbnb's own search work stresses this.
  • The true outcome — a completed, well-rated, re-bookable stay — is sparse and delayed (guests book rarely; the stay is weeks later), so you lean on validated proxy metrics.
  • Some of the highest-impact levers (trust, cancellations) can't be cleanly randomized, so you reach for causal inference.
  • Everything runs on ERF with quality guardrails (cancellations, ratings).

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (4)

A new search ranker wins offline — how do you test it without fooling yourself?Experiment designHardBookings are sparse and the good-stay outcome is weeks away — what do you measure?Metric designHardClick-through is up but bookings and ratings are down — do you ship?Decide from resultsMediumMeasure a host-cancellation program you can't cleanly A/BCausal inferenceHard