A/B Test & Causal Inference Questions

The depth round: randomization units, variance reduction, interference, sequential testing, reading results honestly, and the quasi-experiments (DiD, synthetic control) you reach for when you can't A/B test. For the fundamentals, see Probability & Statistics.

Quick guide

A/B Test & Experimentation Guide

A simple, end-to-end framework for designing and reading an A/B test — hypothesis, metrics, sample size, running it, and the mistakes that fool people.

Read guide →
CompletedCompanyCase QuestionCategory
AirbnbRandomize the guest or the listing? The choice changes your answerFreeTest selection
DoorDashDesign an A/B test for a checkout change — what's the OEC?FreeExperiment design
GoogleWatch time is up but satisfaction is down — do you ship?FreeMetric design
GoogleThe watch-time lift is fading over the days-in window — real or novelty?FreeNovelty & duration
GoogleAverage view duration is a ratio — why can't you just t-test it?FreeVariance & ratio metrics
MetaA Feed change spreads between friends — is your A/B test lying?FreeExperiment design
MetaYour Feed scorecard has 60 metrics — how many 'wins' are fake?FreeMultiple testing
MetaWatch time is up but sends are down — is it a real win?FreeDecide from results
NetflixHow many accounts do you need for a 0.5pp lift?FreePower & sample size
NetflixMeasuring a price change you cannot A/B testFreeCausal inference
AirbnbA new search ranker wins offline — how do you test it without fooling yourself?CompleteExperiment design
AirbnbBookings are sparse and the good-stay outcome is weeks away — what do you measure?CompleteMetric design
AirbnbClick-through is up but bookings and ratings are down — do you ship?CompleteDecide from results
AirbnbMeasure a host-cancellation program you can't cleanly A/BCompleteCausal inference
AirbnbWhy is a user-level A/B biased in a marketplace, and how do you fix it?CompleteExperiment design
AirbnbA time-varying marketplace change — why reach for a switchback?CompleteExperiment design
AirbnbA promo lifts bookings in treated markets — created or just shifted?CompleteDecide from results
AirbnbA price cut fills one calendar by emptying a neighbor's — is your A/B biased?CompleteExperiment design
AirbnbYou can't cleanly randomize a pricing launch — how do you measure it?CompleteCausal inference
AirbnbOccupancy is up but host earnings are down — do you ship?CompleteDecide from results
AirbnbOne price elasticity for all markets — why is that wrong?CompleteHeterogeneous effects

Showing 1–21 of 220