Sony — D2C Payments — A/B Test & Causal Inference Questions

Role context: Data Scientist, D2C Data Science (Payments & Subscriptions) · Est. study time: 65 min · 6 questions

How experimentation works here

Experimentation on payments is unusual:

  • The OEC is net collected revenue, not authorization rate — you can lift the ratio by attempting fewer payments, which collects less.
  • Acquirers, issuers, and fraud systems are shared, so a treatment that routes/retries more can change what the control sees (interference).
  • Many payment changes can't be cleanly A/B'd (routing, a new method, regulation) → holdouts and quasi-experiments.
  • Value is heavy-tailed and some outcomes (chargebacks, hard declines) are rare, so metrics need ratio/variance-aware analysis and rare-event power.
  • Recovery unfolds over weeks (a failed renewal can recover later), so read effects over a proper window.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (6)

Design an experiment for a checkout change — what's the OEC?Experiment designHardYour retry change shares acquirer capacity — is the A/B biased?InterferenceHardMeasure how much a retry/dunning flow actually recoversCausal inferenceHardA routing change shipped country-wide — measure it without an A/BCausal inferenceHardAuthorization rate is a ratio and value is heavy-tailed — analyze it rightVariance & ratio metricsHardApproval is up but chargebacks and cost moved too — do you ship?Decide from resultsHard