A/B Test & Causal Inference Questions

The depth round: randomization units, variance reduction, interference, sequential testing, reading results honestly, and the quasi-experiments (DiD, synthetic control) you reach for when you can't A/B test. For the fundamentals, see Probability & Statistics.

Tutorial guides

A/B Test & Experimentation Guide

A simple, end-to-end framework for designing and reading an A/B test — hypothesis, metrics, sample size, running it, and the mistakes that fool people.

Read guide →
CompanyCase QuestionCategory
AirbnbA new search ranker wins offline — how do you test it without fooling yourself?Experiment design
AirbnbBookings are sparse and the good-stay outcome is weeks away — what do you measure?Metric design
AirbnbClick-through is up but bookings and ratings are down — do you ship?Decide from results
AirbnbMeasure a host-cancellation program you can't cleanly A/BCausal inference
AirbnbWhy is a user-level A/B biased in a marketplace, and how do you fix it?Experiment design
AirbnbRandomize the guest or the listing? The choice changes your answerTest selection
AirbnbA time-varying marketplace change — why reach for a switchback?Experiment design
AirbnbA promo lifts bookings in treated markets — created or just shifted?Decide from results
AirbnbA price cut fills one calendar by emptying a neighbor's — is your A/B biased?Experiment design
AirbnbYou can't cleanly randomize a pricing launch — how do you measure it?Causal inference
AirbnbOccupancy is up but host earnings are down — do you ship?Decide from results
AirbnbOne price elasticity for all markets — why is that wrong?Heterogeneous effects
AmazonForecast a new device with no sales historyNew-product forecasting
AmazonMeasure cannibalization from a launch you can't A/BCausal inference
AmazonIs your forecast good? Evaluate it honestlyForecast evaluation
AmazonEstimate price elasticity without a price experimentElasticity
AmazonMake the portfolio forecast coherent — and decide under uncertaintyPortfolio & decision
AmazonDesign an A/B test for a ranking change — what's the OEC?Experiment design
AmazonRecs get more starts but lower completion — do you ship?Metric design (OEC)
AmazonThe model wins offline but loses online — which do you trust?Decide from results
AmazonEngagement is up week one — but will it retain?Novelty & duration

Showing 1–21 of 158