Watch time is up but satisfaction is down — do you ship? — Google YouTube Data Scientist Interview

Watch time is up but satisfaction is down — do you ship?

Metric designHard

Problem. A ranking experiment lifts watch time +3% (significant), but the satisfaction guardrail — survey satisfaction and "not interested" — moved the wrong way. The team wants to ship on the watch-time win. What's your call?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Is watch time the single success metric, or is satisfaction a pre-declared guardrail we agreed to respect?"
"How big and how significant is the satisfaction drop — a borderline wiggle or a clear move?"
"Is the watch-time gain 'valued' watch time, or could it be sensational/autoplay minutes?"

Lay out your approach

This is the OEC problem. Watch time alone is a gameable objective — YouTube optimizes it with a satisfaction guardrail precisely so a watch-time win that lowers satisfaction doesn't ship.
Don't ship on the primary alone. A watch-time lift with falling satisfaction is the fingerprint of trading quality for quantity (more clickbait/sensational content).
Weigh the net, not the headline. Quantify the satisfaction harm against the watch-time gain, and remember satisfaction leads future watch time and retention.
Name the flip condition — what satisfaction read would make this shippable.

Then check in: "So a watch-time win with a satisfaction drop is the classic OEC trap — I wouldn't ship on the primary alone. Want me to show why watch time alone is gameable and how I'd weigh the net?"

Simulate the back-and-forth

Interviewer

Watch time is up 3% and significant. Why not ship?

You

Because watch time is a quantity metric that can be gamed, which is exactly why YouTube pairs it with satisfaction. A 3% watch-time lift that comes with falling satisfaction usually means the model found more sensational or autoplay-heavy content that holds minutes but annoys people. Shipping it trades long-term trust for short-term minutes — and satisfaction leads future watch time, so this likely costs us later.

Interviewer

The satisfaction drop is small. Doesn't the watch-time win dominate?

You

Not automatically. A "small" satisfaction drop on a huge user base is a lot of dissatisfied sessions, and because it's a leading indicator, it predicts a future watch-time and retention decline that isn't in this experiment's window yet. I'd quantify the expected downstream cost, not treat satisfaction as a rounding error because its p-value is bigger than the watch-time metric's.

Interviewer

So it's a hard no?

You

It's a hold-and-investigate, not necessarily a permanent no. I'd dig into where the watch-time gain and the satisfaction loss come from — if they're in the same content, the model is buying minutes with clickbait and I'd reject it. If the watch-time gain is genuinely valued content and the satisfaction dip is a separate, fixable issue, that changes things. The flip condition is: a version where watch time is up and satisfaction is flat-to-up.

Interviewer

How would you get there?

You

Rebalance the ranking objective to include satisfaction, not just predicted watch time, and re-test. That's the whole reason the OEC blends the two — so the model can't win on minutes by sacrificing quality.

Land a clear conclusion

You

So: don't ship on the watch-time win alone — a lift with falling satisfaction is the OEC trap, buying minutes with quality, and satisfaction leads future watch time and retention. I'd hold, trace whether the watch-time gain and satisfaction loss come from the same content (clickbait), rebalance the objective to include satisfaction, and re-test. It ships only when watch time is up and satisfaction is flat-to-up.

Key summary — what really matters.

This is the OEC problem — watch time alone is gameable, so it travels with a satisfaction guardrail.
Don't ship the primary while a guardrail breaks.
A "small" satisfaction drop is a leading indicator of future watch-time/retention loss.
Trace whether the gain and the loss share content (the clickbait fingerprint).
Flip condition: watch time up and satisfaction flat-to-up, via a rebalanced objective.

Why watch time alone is unsafe. YouTube's own reasoning: ranking by CTR "often promotes deceptive videos the user does not complete ('clickbait')," so it optimizes expected watch time instead (Covington et al., 2016). But raw watch time has its own failure mode — sensational or autoplay-heavy content can hold minutes without satisfying — which is why the practical OEC blends watch time with satisfaction:

$\text{OEC} = \text{watch time} - \lambda \cdot \text{dissatisfaction}$

An experiment that lifts watch time but raises dissatisfaction can lower the OEC even though the headline metric rose. The scorecard:

Metric	Effect	Read
Watch time	+3% (p = 0.01)	clear lift
Survey satisfaction	−0.5pp (p = 0.04)	real drop
"Not interested"	+4%	real rise
OEC (blended)	ambiguous / negative	don't ship on watch time alone

Why the "small" satisfaction move matters. Satisfaction is a leading indicator: dissatisfied viewers churn later, so a satisfaction drop today forecasts a watch-time and retention decline outside this experiment's window. Treating it as noise because its p-value is larger than the watch-time metric's is the mistake — the effect sizes and their downstream leverage matter, not just which p-value is smaller.

The decision. Trace the gain and the loss: if both come from the same newly-boosted sensational content, the model bought minutes with clickbait — reject. Rebalance the objective to penalize dissatisfaction and re-test; ship only when watch time is up with satisfaction intact.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you pin down the causal quantity, the population, and the decision before any math.	You jump to a test or a number without saying what you're estimating or why.	You name the estimand, the unit, the population, and the decision it drives.
Design validity	Whether the design actually identifies the effect — randomization unit, interference, selection/confounding.	You analyze a biased or diluted comparison as if it were clean.	You pick the unit that contains interference, name the threat and its direction, and control it.
Statistical rigor	Whether the stats hold up — power and sample size, variance, the right test, and the assumptions behind them.	You misuse a test, ignore power or variance, or lean on an unchecked assumption.	You size the test, reduce or adjust variance, choose the fitting test, and state what would break it.
Decision & trade-offs	Whether you turn the result into a call — guardrails, cost, and what would flip it.	You stop at "significant / not," or ship on one metric while a guardrail breaks.	You land a ship/hold call, weigh the trade-offs, and name the number that changes it.
Communication	Whether you reason clearly, lead the conversation, and translate the stats for a stakeholder.	You dump jargon, or can't explain the method in plain words.	You reason out loud in order, check in, and give a plain recommendation.

Interview tips

Strong-answer signal: recognizing the OEC trap, refusing to ship the primary while a leading-indicator guardrail breaks, and tracing whether the watch-time gain is clickbait. Common trap: shipping on a significant watch-time lift and waving off a "small" satisfaction drop that actually forecasts future decline.