Apple — AIML Data Operations — A/B Test & Causal Inference Questions

Role context: Data Scientist, AIML Data Operations (Capacity Planning & Analytics) · Est. study time: 65 min · 5 questions

How experimentation and measurement work here

Measuring an annotation operation is distinctive:

  • Quality has no clean ground truth — you estimate it with chance-corrected agreement (Cohen/Fleiss kappa, Krippendorff alpha) and gold questions, not raw percent agreement.
  • Annotators differ in skill and are learning over time, so experiments on task/workflow design are heavily confounded — the right randomization unit and controlling for learning are the whole game.
  • Capacity is a forecasting problem — demand, throughput ramps, overlap, and QA overhead, with scenarios and honest uncertainty.
  • Automation is a measurement problem — evaluate an LLM labeler against gold (per-class recall), and route by confidence on the cost–quality frontier.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

Testing a new guideline — what's the randomization unit?Experiment designHardMeasure label quality when there's no ground truthMeasurement & estimationHardQuality rose after a tool rollout — but annotators were also learningCausal inferenceHardForecast annotation throughput and capacity for planningForecastingHardIs the LLM auto-labeler good enough to ship?Measurement & estimationHard