Randomize the guest or the listing? The choice changes your answer

Test selectionHard

Problem. For a marketplace change, you can randomize on the guest side or the listing side. How do you choose, and how does the choice affect bias and variance?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Which side does the intervention primarily act on — does it change guest behavior, listing visibility, or both?"
"Which side is scarcer in the markets we care about — supply-constrained or demand-constrained?"
"What's the estimand and how much interference does each design leave?"

Lay out your approach

The two designs contain different interference. Guest-side randomization keeps a guest's whole experience consistent but lets treated and control guests compete for the same listings (supply-side interference remains). Listing-side randomization keeps a listing's treatment consistent but lets treated and control listings compete for the same guests (demand-side interference remains).
Match the randomization to the scarce side. Roughly, randomize the side that is not the bottleneck, so the competition that biases you is minimized. In a supply-constrained market, guest-side leaves heavy supply competition; a listing-side or market design may be better, and vice versa.
Weigh variance too. The side with more units gives more power; the sides differ in outcome variance.
When both interfere strongly, escalate to market-level clustering.

Then check in: "Guest-side and listing-side each leave a different interference (supply vs demand competition), so I'd match the design to the scarce side and weigh the variance, escalating to market clusters if both interfere. Want me to reason through a supply-constrained case?"

Simulate the back-and-forth

Interviewer

Give me the intuition for how the two sides differ.

You

Think about what stays "clean." Guest-side randomization gives each guest a consistent experience, so guest-level behavior is well-defined — but treated and control guests still shop the same listings, so if the feature changes how much treated guests book, they consume supply and contaminate control guests (supply-side interference remains). Listing-side randomization gives each listing a consistent treatment, so listing-level effects are clean — but treated and control listings compete for the same guests, so boosting treated listings steals demand from control listings (demand-side interference remains). Neither removes interference; they just move which side leaks.

Interviewer

So how do you choose in a specific market?

You

Match the design to the scarce side. In a supply-constrained hot market, supply is the bottleneck and the dangerous interference is competition for scarce listings — guest-side randomization leaves exactly that, so I'd prefer a listing-side or market-level design there. In a demand-constrained cold market, demand is scarce and listings compete for guests, so listing-side randomization leaves that competition, and a guest-side or market design is safer. The principle: randomize so that the arm boundary doesn't run through the scarce, heavily-competed resource. If both sides are tight, I go to market-level clustering, accepting the power cost.

Interviewer

And the variance side of the decision?

You

There's usually far more of one unit than the other (many guests vs fewer listings, or the reverse in some markets), and their outcomes have different variance and skew — listing-level booking counts are lumpy, guest-level bookings are sparse. So beyond bias, I'd pick the side that also gives adequate power, or use variance reduction. It's a joint bias/variance decision, not bias alone, which is exactly the framing Airbnb's marketplace-experimentation research uses.

Land a clear conclusion

You

So the choice matters because guest-side and listing-side randomization leave different interference — supply competition vs demand competition — plus different variance. I'd randomize the side that keeps the arm boundary away from the scarce, heavily-competed resource (listing/market design in supply-constrained markets, guest-side in demand-constrained ones), weigh the power each gives, and escalate to market-level clustering when both sides interfere. The decision is joint bias and variance, not a default.

Key summary — what really matters.

Each side leaves different interference: guest-side → supply competition; listing-side → demand competition.
Randomize away from the scarce side so the arm boundary avoids the bottleneck.
Supply-constrained → listing/market design; demand-constrained → guest-side.
Weigh variance/power too — it's a joint bias/variance choice.
Escalate to market clustering when both sides interfere.

Which interference remains. Let $I_S$ be supply-side interference (guests competing for listings) and $I_D$ demand-side (listings competing for guests). Roughly:

Design	Clean unit	Interference left	Best when
Guest-side	Guest experience	$I_S$ (supply competition)	Demand-constrained markets
Listing-side	Listing treatment	$I_D$ (demand competition)	Supply-constrained markets
Market cluster	Whole market	Neither (contained)	Both sides tight; costs power

The bias of each design scales with the interference it leaves; pick the design whose remaining interference is smallest given which side is scarce.

Variance. With $n_G$ guests and $n_L$ listings, the side with more independent units and lower outcome variance gives tighter estimates. Listing-level bookings are lumpy (a few listings dominate), inflating variance; guest-level bookings are sparse. The full decision minimizes (bias $^2$ + variance), which is why neither side is a universal default.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you see each side leaves different interference.	"Just randomize users."	Guest-side vs listing-side leave supply vs demand competition.
Design validity	Whether you match design to the scarce side.	Pick a default with no reasoning.	Randomize away from the bottleneck; escalate to market clusters.
Statistical rigor	Whether you weigh variance too.	Bias only.	Joint bias/variance; unit counts and skew.
Decision & trade-offs	Whether the choice is market-specific.	One-size-fits-all.	Supply- vs demand-constrained drives the choice.
Communication	Whether you convey the intuition.	Jargon.	"Keep the arm boundary off the scarce resource."

Interview tips

Strong-answer signal: recognizing that guest-side and listing-side randomization leave supply- vs demand-side interference, matching the design to the scarce side, and treating it as a joint bias/variance decision. Common trap: defaulting to user-level randomization with no awareness that the side you randomize determines which interference biases you.