Whatnot — Customer Experience — A/B Test & Causal Inference Questions

Role context: Data Scientist, Customer Experience (live commerce marketplace) · Est. study time: 75 min · 7 questions

Experimentation in this domain

CX experimentation breaks in a way that is specific and worth naming up front: you cannot randomize the thing you want to study.

You cannot assign users to be scammed. You cannot randomly fail to ship someone's card. So the treatment you actually care about, the bad experience itself, is never randomized, and every comparison involving it is observational and selected on the outcome. What you can randomize is the response: queue priority, channel, refund policy, whether a seller or an agent handles it. That distinction organizes this entire section.

Six things shape everything below:

• The selection is on the outcome. Users who contact support are different from those who do not, in ways correlated with retention. "Users who contacted support retain better" is meaningless: they contacted because they cared enough to stay.
• Resolution direction is confounded with case merit. Cases that get refunded are cases that looked legitimate. Comparing refunded to denied compares valid claims to invalid ones, not policies.
• Policy thresholds are free natural experiments, and most CX orgs have several and use none. A case at $49.99 and one at $50.01 are nearly identical and get different treatment.
• The horizons are long. Retention damage from a bad experience lands over months, so a two-week read is structurally biased toward the cheap policy.
• Interference runs through the audience. A bad experience in a live show is witnessed by everyone watching, so the treated unit is not just the buyer. This is genuinely unusual.
• Sentiment is a construct, not a metric. Auction chat is aggressive by nature. A naive sentiment model will flag the best shows as the worst.

The set covers the impossibility first, then the natural experiment that rescues it, then the one place a real A/B is available (SPS), then sentiment measurement, confounded resolution speed, the horizon problem, and audience interference.