How to Define Metrics — A Systematic Guide

Defining a metric sounds simple and almost never is. A good metric changes what a team decides and does; a bad one quietly sends everyone in the wrong direction. This guide gives you a repeatable, step-by-step way to define metrics that actually drive decisions — the skill product-sense and metrics interviews are really testing.

The seven steps at a glance

1. Start from the decision, not the data (Goal → Signal → Metric).
2. Classify the metric: goal, driver (input), or guardrail.
3. Define it precisely (numerator, denominator, population, window).
4. Pick a North Star and decompose it into a driver tree.
5. Pressure-test it against the properties of a good metric.
6. Add guardrails and counter-metrics.
7. Separate leading vs lagging, and use proxies for delayed outcomes.

Step 1 — Start from the decision (Goal → Signal → Metric)

A metric exists to inform a decision and should have an owner. Google's GSM framework makes this concrete: state the Goal (what success means), the Signal (the observable behavior that indicates success), then the Metric (the precise number that captures the signal).

Goal: "Members find something to watch quickly." → Signal: "They start playing a title soon after opening the app." → Metric: "Median time-to-first-play in a session."

If you cannot name the decision a metric will inform, you do not need the metric yet.

Step 2 — Classify the metric: goal, driver, or guardrail

Borrowing the taxonomy from Trustworthy Online Controlled Experiments (Kohavi, Tang, Xu), every metric plays one of three roles:

Most teams optimize drivers day to day because the goal is slow and noisy, while guardrails keep an optimization honest. A North Star is usually a goal or a very strong driver, surrounded by input metrics.

Step 3 — Define it precisely (the definition checklist)

"Active user" is not a metric — it is an argument waiting to happen. Pin down every metric on these axes:

• Numerator and denominator — what is counted, and over what base.
• Population — which users/accounts/sessions are included or excluded (bots, internal, churned, new vs returning).
• Unit of analysis — user, account, session, or event.
• Time window — daily, weekly, 28-day rolling, per-session.
• Edge handling — ties, nulls, partial periods, time zones.

Vague: "active users." Precise: "accounts with at least one qualifying play of 2+ minutes in a rolling 7-day window, excluding internal and test accounts."

Step 4 — Pick a North Star and decompose it

A North Star Metric (popularized by Sean Ellis and Amplitude's North Star Playbook) is the single measure that best captures the value your product delivers — chosen so that moving it reliably grows the business. Defend the choice in one sentence, then decompose it into a driver tree so you know which levers move it.

North Star: Revenue
   = Paid Members  ×  Average Revenue per Member (ARM)

Paid Members
   = Gross Adds  −  Voluntary Churn  −  Involuntary Churn  +  Reactivations

Each leaf of the tree is a place a team can act, and usually a candidate for its own input metric.

Step 5 — Pressure-test against the properties of a good metric

Before you commit, check the metric against these properties (synthesized from Lean Analytics and the experimentation literature):

• Actionable / behavior-changing — a move in it implies a clear action.
• Comparative — meaningful against a baseline, segment, or time.
• Simple and understandable — a stakeholder can hold it in their head.
• A rate or ratio — normalized, not a raw count.
• Sensitive — it actually moves when the underlying thing changes (so experiments can detect effects).
• Valid — it measures what you claim, with low bias.
• Hard to game — optimizing it does not let someone win on the number while losing on the goal.

Step 6 — Add guardrails and counter-metrics

Every optimization has a way to win the number while harming the business. A complete metric definition names the guardrail that protects against that.

• Optimize ad load (revenue) → guardrail engagement / retention.
• Optimize sign-up conversion → guardrail quality of those sign-ups (do they retain?).
• Optimize recommendation click-through → guardrail long-term satisfaction, not just the next click.

Step 7 — Leading vs lagging, and proxy metrics

Lagging metrics (revenue, annual retention) tell you the truth but arrive too late to steer by. Leading metrics (early engagement, activation) move first and predict the lagging outcome. When the true outcome is delayed or censored, use a proxy / surrogate metric — but only one validated to recover the real outcome, not merely correlated with it.

Framework cheat-sheet

Illustrative example — define success for a new feature

Say a team ships "Continue Watching" reminders.

1. Decision: keep rolling out, iterate, or kill.
2. Goal → Signal → Metric: Goal = members return and resume; Signal = they come back and play a partially-watched title; Metric = 7-day return-and-resume rate among exposed members.
3. Role: this is a driver; the goal is retention; guardrail = notification unsubscribe rate.
4. Precise definition: share of members who received a reminder and played a previously-started title within 7 days, excluding internal accounts, per member.
5. Good-metric check: it is a rate, comparative (vs holdout), behavior-changing, reasonably hard to game.
6. Guardrail: unsubscribe / notification-disable rate must not rise materially.
7. Leading/lagging: return-and-resume (leading) should predict 28-day retention (lagging) — validate that link.

Practice — think it through

Further reading

• Lean Analytics — Alistair Croll and Benjamin Yoskovitz (good-metric traits, vanity metrics).
• Google's HEART and Goal–Signal–Metric frameworks — Kerry Rodden et al., "Measuring the User Experience on a Large Scale."
• North Star Playbook — Amplitude; the North Star concept, Sean Ellis.
• AARRR "Pirate Metrics" — Dave McClure (500 Startups).
• Trustworthy Online Controlled Experiments — Kohavi, Tang, Xu (OEC; goal/driver/guardrail; metric sensitivity and validity).
• Goodhart's Law — on metrics becoming targets.