LLM-as-Judge for Measurement

By the end of this article, you should be able to: explain why LLMs make daily platform-scale measurement feasible, run the golden-set loop that turns a policy into a trustworthy labeler, gate a launch on decision quality, and catch labeler drift after launch.

Step 1: Meet the problem

You want to measure how often content on a platform violates a policy. To do that you need to label a sample of content: violating or not. Two classic options both fall short:

• Human expert labeling is high quality but slow and expensive. Getting tens of thousands of fresh labels per policy can take weeks, which means you can only measure occasionally.
• A simple model-score threshold is cheap but crude, and it grades your own homework if that same score drives enforcement.

Modern LLMs change the economics. A well-prompted LLM can label content about as well as a lightly-trained human moderator, at roughly 100× the throughput and a fraction of the cost — which is what makes daily, platform-representative measurement feasible for the first time (Dobi et al., 2026). But "an LLM said so" is not a measurement. The rest of this article is the governance that makes it one.

Step 2: Policy first, prompt second

The order matters. A subject-matter expert (SME) writes the policy definition — what actually counts as a violation, including edge cases. The prompt is just the operationalization of that policy for the model. If you skip the policy and jump to prompting, you'll bake undocumented judgment calls into the prompt and have no way to audit them.

OpenAI's own content-moderation work follows exactly this order: policy experts write a guideline and a set of labeled examples, and the model reads the policy to label content (OpenAI, 2023).