Define success for a new recommendation model

Measure successMedium

Problem. A team built a new Home recommendation model. Before the launch, they ask you: how should we define and measure whether it's a success?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"What was the model for — more watch time, better new-viewer experience, more diverse recommendations? Success should reflect that."
"Is valued watch time the primary metric, and which guardrails are non-negotiable — satisfaction, creator health, ads?"
"Did it already win offline? If so, I'll treat that as a screen, not proof."

Lay out your approach

Primary metric: valued watch time from a live A/B — the incremental lift vs control, not a gross or offline number.
Satisfaction guardrail. "Not interested," dislikes, survey satisfaction — a watch-time win with rising negatives is a quality regression.
Don't trust offline alone. Offline metrics screen candidates; the online A/B decides, because the two aren't always correlated.
Read over time and by segment. Novelty decay via a days-in view; new/casual/regular viewers separately; creator-side guardrails.

Then check in: "So success is an incremental, sustained valued-watch-time lift from a live A/B, with satisfaction and creator guardrails intact — offline is just the screen. Want me to lay out the scorecard?"

Simulate the back-and-forth

Interviewer

It already beats the old model on offline metrics. Isn't that success?

You

That's a good screen, not a decision. Offline metrics judge the model against logged behavior the old recommender produced, so they can reward a model that just imitates today's system, and they miss how real users react to genuinely new recommendations. YouTube's own experience is that live A/B results aren't always correlated with offline ones. So I'd use the offline win to justify testing, and judge success on the live experiment.

Interviewer

Say the A/B shows watch time up 4%. Ship?

You

Only with the guardrails checked. I'd confirm satisfaction didn't drop — no rise in "not interested" or dislikes — because a watch-time lift can come from more sensational or autoplay-heavy recommendations that people don't actually value. If watch time is up and satisfaction is flat-to-up, that's a real win; if satisfaction fell, it's a hollow number.

Interviewer

Anything about how long you run it?

You

Yes — I'd read the days-in curve for novelty. New recommendations often get an early bump that fades as the novelty wears off, so I'd report the stabilized lift, not the week-one peak, and ramp gradually rather than full-launching.

Interviewer

Who would you expect to move most?

You

Often new and casual viewers, where the model has the most room to improve a thin history. If the win concentrates there and doesn't hurt creators, that's a durable, healthy story.

Land a clear conclusion

You

So: define success as an incremental, sustained valued-watch-time lift from a live A/B — read at the stabilized point, not the peak — with satisfaction and creator-health guardrails intact, and offline metrics used only as a screen. Ship if watch time is up and no guardrail breaks; the number that flips me is rising "not interested" or a lift that fully decays.

Key summary — what really matters.

Success = incremental, sustained valued watch time from a live A/B.
Satisfaction guardrail — a watch-time lift with rising negatives is hollow.
Offline is a screen, not proof — online can disagree.
Read the days-in curve for novelty; report the stabilized lift.
Check segments and creator health, not just the blended number.

The scorecard.

Metric	Role	Illustrative A/B
Valued watch time	Primary	+4% (wk1) → +2.5% stabilized
"Not interested" / dislikes	Guardrail	−3% (good)
Survey satisfaction	Guardrail	+0.3pp
Creator: monetized playbacks	Guardrail	flat
New/casual viewers	Segment	+6%

Why offline can mislead. Offline metrics (precision, recall, ranking loss) score the model against logged impressions generated by the current recommender. A model that best predicts those logs may simply reproduce today's behavior, and it can't observe how users respond to recommendations the old system never showed. YouTube states plainly that "live A/B results are not always correlated with offline experiments" (Covington et al., 2016). So offline is for screening many candidates cheaply; the online A/B is the final call.

Novelty decay. Plot the daily lift over days-in:

Watch-time lift

Illustrative: the daily watch-time lift decays from a novelty peak toward a stable level. Report the plateau (~+2.5%), not the week-1 peak (+4%).

Report the stabilized ~2.5%, not the +4% peak — averaging the whole window overweights the high-novelty early days.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Problem framing	Whether you take a messy business question and break it into clear parts, and say what you think is going on before you dive in.	You jump straight to a metric or a model without understanding the goal, and skip clarifying questions.	You clarify the goal and scope first, lay out a simple structure, and state a clear guess to test.
Methodology and math	Whether your methods, metrics, and numbers are sound, and you can say why you chose them.	You use a heavy method without saying why, make basic math or estimate errors, or forget a control group.	You pick an approach that fits the problem, define the metrics you will judge it on, and handle the obvious edge cases.
Business judgment	Whether you understand why the question matters and tie your answer to real outcomes.	Your idea is technically fine but ignores cost, what is feasible, or what the company actually cares about.	You connect your work to real levers like revenue, growth, or retention, and you name the main risks.
Deployment thinking	Whether you think about how the solution actually runs and lasts in the real world, not just on paper.	You treat it like a class exercise and never say how it would work once it is live.	You think about data quality, scale, monitoring, what happens when it fails, and how you keep it working over time.
Communication	Whether your thinking is easy to follow and you treat the interview as a conversation.	You ramble, lose the thread, or talk at the interviewer instead of with them.	You restate the problem, walk through your logic in order, check in, and sum up your main point at the end.

Interview tips

Strong-answer signal: treating offline wins as a screen, judging success on a live valued-watch-time A/B with a satisfaction guardrail, and reading the days-in curve for novelty. Common trap: shipping on an offline win, or reporting the week-1 peak lift and ignoring decay and the satisfaction guardrail.