Prevalence Estimation (Design-Based Sampling)

By the end of this article, you should be able to: define view-weighted prevalence, explain why a model's own scores can't measure it, design a sample that finds rare violations without biasing the estimate, and say how precise the result is.

What prevalence measures

Prevalence is the share of what people saw that broke a policy. Meta defines it as "the estimated number of views that showed violating content, divided by the estimated number of total content views".1 YouTube publishes the same idea as its violative view rate: for the last quarter of 2020 it was 0.16% to 0.18%, meaning 16 to 18 of every 10,000 views were of content that broke its rules.2

Both count views, not items. One post seen by a million people does far more harm than one seen by nobody, and a metric that counts items would treat them the same. Written out, for items $j$ with $C_j$ views each and a label $Y_j$ that is 1 if the item violates policy and 0 if not:

$\theta = \frac{\sum_j C_j Y_j}{\sum_j C_j}$

The top is views of violating content. The bottom is all views.

Prevalence answers a different question from the enforcement numbers platforms also publish. "Items removed" measures how busy the enforcement system was. Prevalence measures what users were exposed to anyway, including everything the system missed. That's why YouTube says it uses sampling to gain "a more comprehensive view of the violative content we might not be catching with our systems."

Two shortcuts that don't work

The difficulty is that violations are rare and each label is expensive. A label means someone, a trained reviewer or a carefully checked model, reads the item against the policy.

Shortcut one: count what the safety model flags. The platform already has a model scoring every item. Why not report the share of views above its threshold? Because the model can only find what it recognises. Its misses are, by definition, items it scored low. Measuring prevalence with it reports the harm the model already knows about and calls that the total. And when the model improves, the "prevalence" changes even if users saw exactly the same content.

Shortcut two: label a uniform random sample. This is unbiased, but at 0.5% prevalence a sample of 2,000 items holds about 10 violations. An estimate built on 10 positives swings wildly from day to day.

The answer is a third route: sample on purpose where the violations are, then correct for having done so.