OpenAI — Integrity Measurement — Measurement & Modeling Cases

Role context: Data Scientist, Integrity Measurement (Applied Foundations) · Est. study time: 75 min · 6 questions

A different kind of case

These are not product cases in the usual sense. There's no revenue to decompose and no launch to greenlight. The question is always some version of: how do you measure a rare, severe harm precisely enough to trust the number, and to act on it? That makes this a measurement-design archetype, closer to how an epidemiologist estimates disease prevalence or a census bureau designs a survey than to a growth A/B test.

Every strong answer walks the same chain: pin down the estimand → choose a probability sampling design → label with rigor → quantify uncertainty honestly → correct for measurement error → tie the number to a decision. A weak answer reaches for a convenience sample or a raw model threshold and calls it a day.

The methods here are classical and public (probability-proportional-to-size sampling, the Hansen–Hurwitz estimator, Rogan–Gladen label correction), assembled for content safety in two Pinterest KDD 2026 papers and echoed in OpenAI's own moderation writing. The deep dives cite them so you can go to the primary source. Every number below is illustrative.

Each answer is a coaching walkthrough: a Sample answer (clarify the estimand → lay out the design → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example carrying the real math, then a Grading rubric.

Questions (6)

Design an unbiased prevalence estimator for a rare, activity-based harmMeasurement & estimationHardUniform, PPS, or ML-assisted sampling — and what does it cost you?Measurement & estimationHardYour LLM labeler isn't perfect — how does that bias the number, and how do you fix it?Measurement & estimationHardStand up an LLM-as-judge measurement you can actually trustMeasurement & estimationMediumPrevalence is too rare to move in an A/B test — build a metric that canMeasurement & estimationHardThe harm lives partly off-platform — how do you measure it without fooling yourself?Measurement & estimationHard