Design an A/B test for a checkout change — what's the OEC? — DoorDash Consumer & Growth Data Scientist Interview

Design an A/B test for a checkout change — what's the OEC?

Experiment designMedium

Problem. The team ships a streamlined checkout to lift order conversion. Design the A/B test — unit, metrics, and the Overall Evaluation Criterion.

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Decision — ship to all consumers? So the estimand is the effect on orders/retention."
"Randomize by consumer (consistent experience)?"
"Is the risk that faster checkout lifts orders but adds low-quality/mis-orders?"

Lay out your approach

Randomize by consumer. Consistent experience and clean retention reads.
OEC beyond conversion. The primary should be orders (and downstream retention), not just checkout-conversion — a change that lifts conversion but adds cancellations or low-retention orders isn't a win.
Guardrails: cancellations/refunds, support contacts, average order value, and unit economics; plus a marketplace check (a big order lift can strain supply → delivery-time guardrail).
Power and read over enough time for retention, with variance reduction on the noisy order metric.

Then check in: "Consumer-level randomization, an OEC of orders + retention (not just checkout conversion), with cancellation/AOV and marketplace guardrails. Want me to defend the OEC?"

Simulate the back-and-forth

Interviewer

Why not just optimize checkout conversion?

You

Because checkout conversion is a step, not the goal. A faster checkout could lift completed orders by reducing friction — good — or by letting people order impulsively and then cancel, or by pushing through mis-configured orders that get refunded — bad. So I'd make the OEC orders that stick (net of cancellations) and downstream retention, with conversion as a supporting metric. Optimizing the step alone risks shipping a change that games the funnel without creating real, retained orders.

Interviewer

What guardrails, and why the marketplace one?

You

Cancellations/refunds and support contacts (did friction move to after checkout), average order value (did faster checkout shrink baskets), and unit economics. The marketplace guardrail matters because a large conversion lift means more demand, which in a supply-constrained market raises delivery times and costs — so I'd watch delivery-time and fulfillment guardrails, especially in tight markets. A consumer win that degrades fulfillment is a marketplace loss.

Interviewer

Anything about reading it?

You

Order rate is noisy and retention is delayed, so I'd run long enough for a retention read and use CUPED (covariate adjustment on pre-period ordering) to cut variance and reach power faster. I'd also check the split is clean (no sample-ratio mismatch) and read guardrails before declaring a win.

Land a clear conclusion

You

So: consumer-level randomization; an OEC of orders net of cancellations plus retention, not checkout conversion alone; guardrails on cancellations, AOV, unit economics, and delivery-time (marketplace); powered with CUPED and read over a retention window. Ship only if real, retained orders rise with guardrails intact.

Key summary — what really matters.

Randomize by consumer; consistent experience, clean retention.
OEC = orders (net of cancellations) + retention, not checkout conversion alone.
Guardrails: cancellations/refunds, AOV, unit economics, delivery-time (marketplace).
CUPED for the noisy order metric; read over a retention window; check SRM.

Conversion up, orders flat? Two reads of a checkout change:

Signal	Story A (real)	Story B (hollow)
Checkout conversion	+5%	+5%
Cancellations	flat	+8%
Net orders	+4.5%	+1%
D30 retention	+0.5pp	−1pp

Same conversion lift; Story A is a real win, Story B just added friction after checkout. Only the net-orders and retention read distinguishes them — which is why the OEC can't be checkout conversion.

CUPED. Adjusting the order metric by each consumer's pre-period order rate removes much of the between-consumer variance, often cutting the needed sample by a large factor — essential for a noisy metric like orders.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether the OEC is orders + retention.	Checkout conversion.	Orders net of cancellations + retention.
Design validity	Whether guardrails cover the risks.	None.	Cancellations, AOV, unit economics, delivery-time.
Statistical rigor	Whether you handle power/noise.	Ignore variance.	CUPED; retention window; SRM check.
Decision & trade-offs	Whether you catch the marketplace effect.	Consumer metrics only.	Supply strain from a demand lift.
Communication	Whether you justify the OEC.	"Conversion up."	OEC rationale + guardrails.

Interview tips

Strong-answer signal: an orders+retention OEC (not checkout conversion), the right guardrails including marketplace supply, and CUPED for power. Common trap: optimizing checkout conversion alone, ignoring cancellations, retention, and supply strain.