Stripe — Experimental Projects — A/B Test & Causal Inference Questions

Role context: Data Scientist, Experimental Projects (zero-to-one product incubation) · Est. study time: 75 min · 7 questions

Experimentation in this domain

Experiments on a new Stripe product look nothing like a consumer A/B test with millions of users. The thing you're trying to learn about is businesses, and a pilot has a few dozen of them.

• The unit is the hard part. Randomizing businesses gives you 9 against 9. Randomizing buyer sessions inside each business gives you thousands of sessions, but they're clustered, and the answer you care about (will the next business benefit?) is still limited by how many businesses you have.
• Stripe's own tools show the tricks. Stripe's payment-method A/B tool hashes the buyer and a time window so small merchants reach significance faster, and it counts only sessions where the treatment can actually show. Its large payment-method study ran a power analysis, then an A/A test, then a pilot, before the real experiment. Those habits matter even more on a prototype, where event joins are new and often broken.
• Heavy tails. One large business can be most of the revenue. Test choice and the estimand matter.
• Opt-in adoption. Businesses choose whether to switch a feature on, so the people who use it aren't random. Randomizing the invitation is the standard way out.
• Speed matters. The team wants answers in weeks, so decisions use intervals, expected loss, and planned interim looks, not a single fixed-horizon p-value.
• Models come with prototypes. A risk score that blocks payments creates its own measurement problem: blocked payments never show whether they were fraud.

This zero-to-one role weights small-sample design and power (where businesses, not sessions, set the precision), decision-making under wide intervals, opt-in causal inference, and model threshold evaluation. It skips variance-heavy topics like large-scale multiple testing, which matter more on mature products with hundreds of metrics.