Average view duration is a ratio — why can't you just t-test it? — Google YouTube Data Scientist Interview

Average view duration is a ratio — why can't you just t-test it?

Variance & ratio metricsHard

Problem. You want to compare average view duration (AVD = total watch time ÷ total views) between two arms, randomized by user. A teammate computes AVD per arm and runs a standard two-sample t-test. Why is that variance estimate wrong, and what do you do instead?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Is randomization by user, but AVD is computed per view? That mismatch is the whole issue."
"Are we after the effect on AVD (a ratio of two totals), not a simple per-user mean?"
"Do heavy users contribute many views? Then per-view observations within a user are correlated."

Lay out your approach

Name the mismatch: ratio metric with a coarser randomization unit. AVD is watch time ÷ views, but users are randomized, and one user contributes many correlated views. Treating views as independent understates the variance.
The naive t-test on per-view data is wrong because views within a user aren't independent — it undercounts variance and inflates false positives.
Use the delta method (or a bootstrap/clustered variance at the user level) to get the correct variance of the ratio.
This is the standard fix for ratio metrics whose numerator and denominator are both random and correlated.

Then check in: "So AVD is a ratio with a user-level randomization unit — the per-view t-test understates variance. I'd use the delta method or user-level bootstrap. Want me to write out the delta-method variance?"

Simulate the back-and-forth

Interviewer

AVD is just a mean of durations. Why not a normal t-test?

You

Because AVD isn't a mean over independent units here — it's a ratio of two totals, watch time over views, and we randomized by user, not by view. One user contributes many views, and those views are correlated (a user who watches long does it repeatedly). If I treat each view as independent, I pretend I have far more independent observations than I do, so the variance is understated and the test calls things significant too often.

Interviewer

What's the right variance?

You

The delta method. AVD is a function of two random totals — mean watch time and mean views per user — so I linearize the ratio and combine the variances of the numerator and denominator, including their covariance. Equivalently, I can bootstrap at the user level (resample users, not views) or use a cluster-robust variance clustered on user. All three respect that the user is the independent unit.

Interviewer

Does it matter much in practice?

You

Yes — ignoring the within-user correlation can badly understate the variance, so a "significant" AVD change can be noise. The correction often widens the CI substantially. It's the classic randomization-unit-coarser-than-analysis-unit problem, and getting it wrong is a common way teams ship false AVD wins.

Interviewer

Any simpler alternative?

You

Compute a per-user metric instead — e.g., average watch-time-per-view within each user, then a t-test across users. That makes the user the unit of analysis and sidesteps the ratio issue, though it weights users equally rather than by views. Which is right depends on whether we care about the per-view or per-user quantity; if AVD (per-view) is the metric of record, the delta method is the honest tool.

Land a clear conclusion

You

So: AVD is a ratio of two random, correlated totals with a user-level randomization unit, so the per-view t-test treats correlated views as independent and understates the variance. I'd use the delta method (or a user-level bootstrap / cluster-robust variance clustered on user) to get the right CI. It's the randomization-unit-coarser-than-analysis-unit trap, and skipping it is how a noisy AVD change gets shipped as a win.

Key summary — what really matters.

AVD is a ratio (watch time ÷ views), not a mean over independent units.
Randomization is by user; views within a user are correlated — the per-view t-test understates variance.
Use the delta method for the ratio's variance (numerator, denominator, covariance).
Or bootstrap / cluster at the user level — the user is the independent unit.
This is the randomization-unit-coarser-than-analysis-unit trap.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you pin down the causal quantity, the population, and the decision before any math.	You jump to a test or a number without saying what you're estimating or why.	You name the estimand, the unit, the population, and the decision it drives.
Design validity	Whether the design actually identifies the effect — randomization unit, interference, selection/confounding.	You analyze a biased or diluted comparison as if it were clean.	You pick the unit that contains interference, name the threat and its direction, and control it.
Statistical rigor	Whether the stats hold up — power and sample size, variance, the right test, and the assumptions behind them.	You misuse a test, ignore power or variance, or lean on an unchecked assumption.	You size the test, reduce or adjust variance, choose the fitting test, and state what would break it.
Decision & trade-offs	Whether you turn the result into a call — guardrails, cost, and what would flip it.	You stop at "significant / not," or ship on one metric while a guardrail breaks.	You land a ship/hold call, weigh the trade-offs, and name the number that changes it.
Communication	Whether you reason clearly, lead the conversation, and translate the stats for a stakeholder.	You dump jargon, or can't explain the method in plain words.	You reason out loud in order, check in, and give a plain recommendation.

Interview tips

Strong-answer signal: recognizing AVD as a ratio with a user-level randomization unit, and using the delta method (or user-level bootstrap / clustered variance) instead of a per-view t-test. Common trap: running a standard t-test on per-view data, treating correlated within-user views as independent and understating the variance.