NVIDIA — GeForce NOW — A/B Test & Causal Inference Questions

Role context: Data Scientist, Voice of the Customer (GeForce NOW) · Est. study time: 75 min · 7 questions

Experimentation in this domain

Experimentation on GeForce NOW splits cleanly in two, and knowing which half you are in is most of the skill.

Client-side changes can be randomized. An encoder setting, a bitrate ladder, a client build: you can assign players to arms and read a clean comparison. These are real A/B tests and the usual rules apply.

Infrastructure changes cannot. A new data center, a routing change, or added capacity in a region hits everyone routed there. There is no control group, because you cannot randomize which region a player lives in. So a large share of the most important work at GeForce NOW is quasi-experimental by construction, and difference-in-differences and synthetic control are working tools here rather than exam topics.

Five things shape every question below:

• The metrics are quantiles, and quantiles are awkward. The product is ruined by the worst moments, so the metric of record is p99 latency, not mean latency. You cannot run a t-test on a p99. The variance of a quantile depends on the density of the data at that quantile, and in the tail the density is thin, so tail metrics are far noisier than their sample size suggests.
• Sessions inside a player are heavily correlated. Same house, same router, same ISP, same TV. Treating sessions as independent is the fastest way to a wrong p-value here, and the correlation is much stronger than most people guess.
• The signal has a 5x daily cycle. Evening peak against overnight trough. NVIDIA's own published position is that "cloud KPIs surge during traffic spikes, fall quiet at night, and spike again during batch jobs," which is why static thresholds and short baselines both fail.
• Heterogeneity is the finding, not a footnote. A change that helps at 100 Mbps can hurt at 25 Mbps, and the average of those two is a number describing nobody. The players who churn are disproportionately in the segment the average hides.
• Labels are sparse and unreliable. You have ground truth only for incidents someone noticed and filed. Any supervised approach learns to find the problems you already catch.

This is a measurement and machine learning role rather than a product-experimentation role, so the set below covers the experiment-design core (design, power, test selection, reading a result) and then weights depth toward heterogeneous effects, causal inference without randomization, threshold-setting under non-stationarity, and multiple testing at monitoring scale, which is where this job actually lives.