Anthropic — Developer Productivity — A/B Test & Causal Inference

Role context: Data Scientist, Developer Productivity · Est. study time: 55 min · 5 questions

How these map to the role

These are the causal problems of measuring an internal, small-N, contested outcome: experiments on tooling, causal AI-impact under voluntary adoption, power with few engineers, surrogate metrics, and not fooling yourself. Each answer is a coaching walkthrough: a Sample answer (clarify → approach → simulated back-and-forth → clear call), a Deep dive with the math, then a Grading rubric.

The recurring instinct: instrument objectively, identify causally, respect small-N uncertainty, and hold conclusions loosely.