OpenAI — Integrity Measurement — Product Sense & Metrics

Role context: Data Scientist, Integrity Measurement (Applied Foundations) · Est. study time: 40 min · 5 practice questions

What this role actually is

Most "product sense" prep imagines a consumer product with a signup funnel and a revenue number. This role is different, and the difference is the whole point. There is no funnel to optimize. The "product" you own is a measurement system: a pipeline that turns a safety policy into a trustworthy number for how much of a rare, severe harm is happening on OpenAI's platforms, and how well the response to it is working.

OpenAI frames safety as core to its mission and leans hard on one principle in particular: rigorous measurement. In its own words, "safety research requires standardizable measurement… being able to quantify risks effectively guides research direction and prioritization," and evaluations "start with measurement goals, often guided by a threat model" (OpenAI, "How we think about safety and alignment"). The Applied Foundations team defends the platform against abuse and scaled attacks; the Integrity pillar owns the systems that find and respond to bad actors. As those enforcement systems mature, the pressing question becomes: are they working, and how much harm is left? Answering that, robustly enough to set goals and report externally, is this job.

So the mental model to bring is closer to a public-health epidemiologist estimating disease prevalence, or a census bureau designing a survey, than a growth analyst reading an A/B dashboard. The craft is design-based sampling, estimation under measurement error, and using LLMs as scalable labelers.

The harms being measured

The harms here are severe, rare, and often about behavior rather than a single piece of content. The distinction between an activity-based and a content-based problem is one this team cares about a great deal, and it is worth being precise about: a content harm is "this image violates policy"; an activity harm is "this network of accounts is running a coordinated influence operation." The second is a pattern of behavior, which is far harder to sample and label than one image.

From OpenAI's public writing, the verticals a measurement DS might own include:

Every one of these is a low base rate (rare), a high cost per instance (severe), and often network-shaped. That combination is exactly what makes the measurement statistically hard, and why the team needs a specialist.

The measurement journey

Instead of a user funnel, the journey to map is the life of a measurement: how a policy definition becomes a daily number you can trust. In practice, a prevalence pipeline like this has four stages.

Two design choices in that pipeline are worth internalizing now, because interview questions circle back to them constantly:

• You sample by design, not by convenience. Because violations are rare, uniform sampling burns the label budget on obviously-safe content. You draw a probability sample weighted toward high-exposure and high-risk items, then reweight by the known sampling probabilities so the estimate stays unbiased. This is the difference between a survey and an anecdote.
• The thing that flags content is not the thing that grades it. The production enforcement model's scores are used only to prioritize what to review, never as the label. Grading your enforcement with its own scores is "grading your own homework," and it systematically makes you look better than you are (the selective-labels problem). The LLM labeler is kept independent on purpose.

Metrics by domain

A measurement DS owns a small, sharp set of numbers. They split into three groups: how much harm there is, how good the response is, and how much you can trust the measurement itself.

How much harm (the top-line):

• Exposure-weighted prevalence — the fraction of impressions (views) that landed on violating content, not the fraction of items. One item can get one view or a million, so weighting by impressions reflects what users actually experienced: $\theta = \frac{\sum_j C_j Y_j}{\sum_j C_j}$ where $C_j$ is impressions to item $j$ and $Y_j$ is its violation label.
• User reach — the fraction of daily active users who saw at least one violating impression. Prevalence and reach can diverge: a small amount of content hitting many users is a different problem than a lot of content hitting a few.

How good the response is:

• Recall / catch rate — of the true violations out there, what share did enforcement catch? This measures response quality, which is invisible in prevalence alone.
• Precision of enforcement — of the items enforcement acted on, what share truly violated? False positives here mean punishing good users.

How much you can trust the number:

• CI half-width and effective sample size (ESS) — how precise the estimate is. A prevalence number with no interval is not a measurement, it's a guess.
• Labeler sensitivity $r$ and false-positive rate $f$ — how good the LLM labeler is against a human-labeled gold set. These gate the whole metric and feed the label-error correction.
• Minimum detectable effect (MDE) — the smallest change the metric can reliably catch. If the MDE is bigger than the change you care about, the metric can't support a goal or an A/B test.

Expect to be asked how you would build a metric that can carry a goal or an A/B test when prevalence itself cannot. That is a real, recurring problem on this team: when base rates are tiny, absolute prevalence is too noisy to move in an experiment, so you need a surrogate metric (a calibrated score-bucket estimate) and delta-focused inference instead.

The data you'd look at daily

The tables here look nothing like a product analytics job, because the thing being built is an estimate rather than a funnel. Three do most of the work (example columns and rows):

1. The daily sample — one row per sampled item, with everything needed to weight it back up:

Three columns are the craft of the role. risk_score decides who gets looked at and never becomes the label — that separation is what stops the measurement from grading its own homework. sample_prob is the probability that item was drawn, and weight is its inverse: item four stands in for a thousand others, which is how a sample of tens of thousands can describe a population of billions.

That last row is also the danger. A weight of 1000 means one labelling mistake on that row moves the whole estimate, which is why effective sample size matters more than raw sample size here.

2. Labeler calibration against the gold set — one row per policy, refreshed on a fixed cadence:

This table gates every number the team publishes. A labeler that catches 62% of true cases and falsely flags 3.1% is not measuring prevalence, it is measuring itself, until the estimate is corrected for both. And drift_30d is why the gold set gets refreshed rather than built once: the covert-influence labeler has lost nine points in a month, most likely because the behaviour changed underneath a prompt that did not.

3. The published estimate — one row per policy × day, and the only table anyone outside the team sees:

Read the last row and the last column together. Effective sample size collapsed to 9,100 from the same 48,000 labels because the weights were spiky, and the minimum detectable effect is 51% — meaning this metric cannot see anything smaller than a halving. Reporting it as a number without that caveat, or setting a goal against it, is the mistake this role exists to prevent.

What the dashboards look like at OpenAI

A mature measurement dashboard is not a single line. It's a prevalence trend with confidence bands, drill-downs, and health diagnostics, all from one daily sample.

Prevalence trend (illustrative): a 7-day smoothed prevalence line per policy, with a shaded 95% CI band and vertical markers for interventions (a policy change, a new enforcement model). The band is the honest part: it's what tells a VP whether last week's dip is real or noise.

Drill-downs from one sample: the same daily sample re-sliced by surface, viewer geography, or content age, so a team can localize a change without commissioning a new study. This "one global sample, many pivots" property is a core design goal.

Health diagnostics: CI half-width, ESS vs raw sample size (a big gap warns of heavy weights), sample positive rate, and the tracked labeler quality $(r, f)$ over time. When the metric wobbles, these tell you whether the cause is sampling noise, labeler drift, or a real shift in the world.

All of this ultimately feeds decisions: a goal ("cut prevalence of policy X by 20% this half"), a guardrail (page on-call if it jumps), an escalation (a new harmful pattern to investigate), or an external report. A measurement that doesn't drive one of those isn't done.

Quick check

Practice Questions