Qualified play-through is a ratio metric — analyze it correctly

Variance & ratio metricsHard

Problem. Your key metric is qualified play-through rate (qPTR = engaging plays / impressions). Someone runs a two-sample t-test on per-user qPTR and reports p = 0.01. What's wrong, and how do you analyze it properly?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Your nav change touches every surface — is a user-level A/B valid?How many users (or markets) do you need to detect a 1pp lift? →