Did the paid-sharing crackdown actually cause the revenue lift?

Measure impactHard

Problem. Netflix rolled out the paid-sharing crackdown (borrower prompts plus Extra Member), and in rolled-out regions revenue rose. Leadership wants to credit the crackdown with the full lift. Did it cause the lift, and how would you estimate the true incremental effect?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

First pin down what the prompt is really asking:

"How is the lift measured — total revenue, ARM, or memberships — and over what window?"
"When leadership says the crackdown caused it, do they mean the full before/after change, or the incremental effect over what would have happened anyway?"
"By the crackdown do you mean the whole bundle — borrower prompts plus Extra Member — as one thing?"

Lay out your approach

Is it causal? A before/after in treated regions also captures everything else moving over those months — content, seasonality, price, FX — and rollout regions were likely chosen non-randomly (highest sharing), which biases the naive number up.
Use the assignment mechanism. A staggered regional rollout is a natural experiment: not-yet-treated regions are controls.
Method from the mechanism. Difference-in-differences (treated vs not-yet-treated), or synthetic control per region (a weighted blend of untreated regions as the counterfactual).
Name the threatened assumption. Parallel pre-trends — treated and control must have moved together before rollout; if not, switch to synthetic control.
Report incremental, not raw — with a confidence interval, not the before/after number.

Then check in: "Shall I start with how I'd build the counterfactual, or with the assumption it rests on?"

Simulate the back-and-forth

Interviewer

Leadership wants to credit the crackdown with the whole lift. Do you buy it?

You

Not the raw before/after. Over those months, content, seasonality and FX also moved, and we probably rolled out to the highest-sharing regions first, so the naive number is biased up. To isolate the crackdown I'd need a comparison — was it rolled out everywhere at once, or staggered?

Interviewer

Staggered, region by region. What do you do with that?

You

That's a natural experiment. I'd use the not-yet-treated regions as controls and run difference-in-differences, or build a synthetic control — a weighted blend of untreated regions — for each treated region. The causal lift is the gap above that counterfactual, not the raw rise.

Interviewer

Treated regions clearly rose more, though. Isn't that the impact?

You

Only if the controls were flat. If untreated regions also rose on the same content and seasonality, the crackdown's real effect is the difference between the two — usually a good bit smaller than the headline.

Interviewer

What could make that estimate wrong? What do you think?

You

Non-parallel pre-trends. If we rolled out to regions already growing faster, the lines weren't parallel before rollout and DiD overstates it. I'd test the pre-period explicitly; if it fails, I'd trust synthetic control, which matches the pre-period by construction.

Land a clear conclusion

You

So I'd report the crackdown's incremental lift as the gap above comparable regions, with a confidence interval — not the raw before/after, which is meaningfully bigger. The whole estimate hinges on parallel pre-trends; if the treated regions were already diverging before rollout, I trust the synthetic-control number over plain DiD, and I'd hand the formal parallel-trends and placebo tests to the experimentation review.

Key summary — what really matters.

Don't credit the before/after — it's confounded with content, seasonality, FX, and region selection.
Use the staggered rollout as a natural experiment (DiD / synthetic control).
The causal lift is the gap above the counterfactual, not the raw rise.
It rests on parallel pre-trends — test them, else synthetic control.
Report an incremental number with a range, not a point.

Naive vs causal (illustrative).

Estimate	Revenue lift attributed
Naive before/after (treated regions)	+9%
Difference-in-differences (treated − control)	+5%
Overstatement from confounds / selection	4 pts

The logic: treated regions went 100 → 109 (+9%), but the not-yet-treated control regions went 100 → 104 over the same window (content, seasonality). The crackdown's incremental effect is the difference of differences: 9 − 4 ≈ 5%.

Treated regionSynthetic counterfactual

Illustrative: treated region vs its synthetic counterfactual (a weighted blend of not-yet-treated regions). They track before rollout, then diverge — the gap after is the causal lift.

The assumption that decides it. DiD is only valid if treated and control regions had parallel pre-trends. If rollout targeted regions already growing faster, the lines weren't parallel and DiD is biased upward — that's when you switch to synthetic control, which builds a counterfactual that matches the pre-period by construction. The execution (formal parallel-trends test, placebo-in-time) lives in the Stats & A/B section.

Recommend. "Report the crackdown's incremental lift as ~5% with a CI, not the raw 9%. The single thing that would change the estimate is non-parallel pre-trends — if the treated regions were already diverging before rollout, I'd trust the synthetic-control number over DiD."

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Problem framing	Whether you take a messy business question and break it into clear parts, and say what you think is going on before you dive in.	You jump straight to a metric or a model without understanding the goal, and skip clarifying questions.	You clarify the goal and scope first, lay out a simple structure, and state a clear guess to test.
Methodology and math	Whether your methods, metrics, and numbers are sound, and you can say why you chose them.	You use a heavy method without saying why, make basic math or estimate errors, or forget a control group.	You pick an approach that fits the problem, define the metrics you will judge it on, and handle the obvious edge cases.
Business judgment	Whether you understand why the question matters and tie your answer to real outcomes.	Your idea is technically fine but ignores cost, what is feasible, or what the company actually cares about.	You connect your work to real levers like revenue, growth, or retention, and you name the main risks.
Deployment thinking	Whether you think about how the solution actually runs and lasts in the real world, not just on paper.	You treat it like a class exercise and never say how it would work once it is live.	You think about data quality, scale, monitoring, what happens when it fails, and how you keep it working over time.
Communication	Whether your thinking is easy to follow and you treat the interview as a conversation.	You ramble, lose the thread, or talk at the interviewer instead of with them.	You restate the problem, walk through your logic in order, check in, and sum up your main point at the end.

Interview tips

Strong-answer signal: leading with the assignment mechanism (staggered rollout), naming selection bias and its direction, and separating the causal estimate (DiD / synthetic control) from the raw before/after. Common trap: accepting the before/after lift as the program's impact, or jumping to a fancy method without justifying it from how treatment was assigned. Trains: non-randomization; confounder-direction.