OpenAI — Applied Product — A/B Test & Causal Inference Questions

Role context: Data Scientist, Applied Product · Est. study time: 60 min · 5 questions

How experimentation works for an LLM product

Testing ChatGPT is subtle in ways a normal UX A/B isn't:

  • A model change moves everything at once (quality, tone, latency, cost), so an experiment compares an entangled bundle and the Overall Evaluation Criterion is contested.
  • The core outcome — helpfulness — is fuzzy, so you lean on proxies (regeneration, corrections, task completion) and must validate them.
  • Easy proxies can be gamed: OpenAI's own sycophancy rollback showed that optimizing short-term approval (thumbs-up) can degrade real quality, and qualitative review caught what the metrics missed.
  • Offline evals screen models but don't always predict online behavior, and short-term engagement can diverge from long-term retention/trust.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Design an A/B test for a new model version — what's the OEC?Experiment designHardThe update gets more thumbs-up — but is it just sycophancy?Metric designHardIt wins on offline evals but loses online — which do you trust?Decide from resultsMediumEngagement is up week one — but will it retain?Novelty & durationHardOnly 2% of users rate answers — is your quality metric even real?Metric designHard