Pinterest — Trust & Safety — A/B Test & Causal Inference Questions

Role context: Data Scientist, Trust & Safety (prevalence measurement) · Est. study time: 80 min · 7 questions

Experimentation in this domain

This section looks different from every other one on this site, and the difference is the point.

Most data science roles run experiments to learn whether a change worked. This role's primary statistical problem is estimating a rare quantity accurately and cheaply, and most of the interventions it evaluates cannot be randomized at all. A policy change, an enforcement sweep, or a classifier retrain ships platform-wide. So the toolkit shifts: sampling theory becomes the main subject rather than a footnote, and quasi-experimental reads replace A/B tests for most questions.

Six things shape everything below:

• Violations are rare and labels cost money. At a 0.01% rate, uniform sampling needs roughly a million labels to see a hundred positives, for one policy on one day. That single fact forces every design choice that follows.
• The fix is deliberate bias, then correction. Sample disproportionately from high-exposure and high-risk content, then reweight by the inverse sampling probability. The sample is biased on purpose; the estimate is not.
• The risk score prioritizes; it never labels. Let the classifier's output become the label and prevalence measures your own thresholds. Retune the model and safety "improves" with nothing changing. Pinterest calls this failure lensing, and design-based reweighting is what removes it.
• More positives can mean less precision. Lean too hard on the risk score and the weights get spiky, which destroys effective sample size. You can double your positive rate and widen your confidence interval. Effective sample size, not positives found, is the thing you tune.
• The labeler is part of the instrument. An LLM with a small false-positive rate, applied to an enormous negative population, can overstate a rare prevalence by an order of magnitude. Label error is not a caveat here, it is a bias term you must correct.
• The metric's definition can move. The LLM prompt is the operational definition. A policy clarification changes the number with nothing changing on the platform. No other domain has this failure mode so sharply.

The set below covers the measurement core (sampling design, the estimator, label quality, and power), then weights depth toward experiment design where randomization is possible, causal reads where it is not, and multiple testing at monitoring scale. Question ab5 is the only true A/B test in the set, and that ratio is honest to the job.