Watch time is up but sends are down — is it a real win? — Meta Instagram Data Scientist Interview

Watch time is up but sends are down — is it a real win?

Decide from resultsHard

Problem. A Reels ranking experiment lifts watch time +4% (significant), but sends per reach (DM shares) fell −3%. The team wants to ship on watch time. What's your call?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Are both moves significant, and what are their intervals? A −3% sends move might be noise or real."
"What's our stated objective — is watch time the primary and sends a guardrail, or are both first-class?"
"Is the watch-time gain broad, or driven by a specific content type (e.g., longer, lower-quality Reels)?"

Lay out your approach

Understand why the two disagree. Watch time and sends measure different things: time is passive consumption, a send is a costly, high-intent endorsement that predicts reach to new people. If time is up but sends are down, the ranking may be surfacing stickier-but-less-shareable content — more scrolling, less "this is worth sharing."
Decide which signal is closer to long-run value. Sends are the signal Instagram emphasizes precisely because they resist gaming and predict healthy growth. A watch-time win bought by falling sends may be shallow engagement.
Treat sends as a guardrail with a threshold, not a metric to net against time by size.
Diagnose the content mix driving the watch-time lift before deciding, and check integrity ("not interested").

Then check in: "Watch time and sends disagree, which suggests the ranking traded shareable content for sticky content. I'd treat sends as a guardrail and diagnose the content mix before shipping. Want me to walk the trade-off?"

Simulate the back-and-forth

Interviewer

Watch time is our headline and it's clearly up. Why hesitate over sends?

You

Because watch time is the more gameable of the two, and sends are the check on exactly the failure it's prone to. A ranking can lift watch time by showing longer or more hypnotic content that people passively watch but don't value enough to share. Sends dropping is the tell that we may have done that — we bought passive minutes and lost the high-intent signal that predicts reaching new audiences. So I don't want to ship a watch-time number that a falling sends number is warning me is hollow.

Interviewer

What if the sends drop is small and barely significant?

You

Then I weigh it against the asymmetry and the mechanism. Sends are sparser, so a real drop is harder to detect — a barely-significant −3% could still reflect a genuine shift. I'd want to know why it moved: if the watch-time lift comes from a content type that's inherently less shareable, the sends drop is causal and worrying; if sends fell for unrelated reasons, less so. I'd get a more powered read on sends specifically rather than dismiss it, because it's the guardrail against the exact way watch time misleads.

Interviewer

So what's the ship decision?

You

If the watch-time gain is broad and sends recover or hold once I fix the content mix, ship. If the gain is concentrated in longer, low-share content and sends stay down, I'd not ship as-is — I'd try to keep the watch-time gain while protecting sends (down-weight the low-share content driving the drop) and re-test. The decision rule is watch time up and sends not materially down, not watch time up alone.

Land a clear conclusion

You

My call: don't ship on watch time alone. Sends are the guardrail against shallow watch-time gains, so I'd diagnose whether the ranking traded shareable content for sticky content, get a properly powered read on the sends drop, and look for a variant that keeps the watch-time lift with sends protected. Ship only when both the objective and the quality signal agree — a watch-time win contradicted by falling sends is exactly the case Instagram's metric design is built to catch.

Key summary — what really matters.

Watch time and sends measure different things — passive time vs high-intent sharing.
Sends guard against the way watch time is gamed (longer, stickier, less valuable content).
Treat sends as a guardrail, not a metric to net against time by magnitude.
Diagnose the content mix driving the watch-time lift; get a powered read on sends.
Ship rule: watch time up and sends not materially down.

Why sends catch shallow wins. Model per-impression value as watched value plus sharing value:

$\text{value} = w \cdot \text{watch\_seconds} + s \cdot \text{sends}$

A ranking that raises watch_seconds but lowers sends can leave total value flat or down even with the watch-time headline up. Suppose:

Signal	Control	Treatment	Weight (illustrative)
Watch seconds / impression	20.0	20.8 (+4%)	1.0
Sends / 1k reach	30	29.1 (−3%)	15

Watched value: $+0.8$ per impression. Sharing value: $15 \times (29.1-30)/1000 \cdot (\text{impressions})$ — because sends are heavily weighted (they drive new-audience reach), even a small percentage drop can offset much of the watch-time gain. The exact netting depends on the weights, which is why you don't net raw percentages — you diagnose the mechanism.

Power note on sends. Sends are a rarer event than watch-seconds, so their standard error is larger; a −3% sends effect may need more traffic to confirm than a +4% watch-time effect. "Not significant yet" on the guardrail isn't "safe."

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you know what each signal captures.	You treat watch time as the only metric.	You explain passive time vs high-intent sends.
Design validity	Whether you diagnose the mechanism.	You read the two numbers in isolation.	You trace the watch-time lift to a content-mix shift.
Statistical rigor	Whether you handle sends' higher variance.	"Sends drop isn't significant, ignore it."	You note sparse-event power and get a stronger read.
Decision & trade-offs	Whether you protect the quality signal.	Ship on watch time.	Seek a variant that keeps time with sends protected.
Communication	Whether the call is crisp and defensible.	"Mixed results."	"Ship only if watch time up and sends not materially down."

Interview tips

Strong-answer signal: explaining why sends guard against shallow watch-time gains, treating sends as a guardrail, and diagnosing the content mix before deciding. Common trap: shipping on the watch-time headline and dismissing the sends drop as noise without considering its higher variance.