Measuring a price change you cannot A/B test

Causal inferenceHard

Problem. Netflix raises the Standard price in one country. You cannot randomize price across users. How do you estimate the causal effect on revenue and churn?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

Pin down the setup before solving:

"Is the price change rolled out to the whole country at once, and can we not randomize users at all?"
"Do we have comparable markets that didn't change price, with clean pre-period data?"
"Are we measuring revenue, churn, or the net — and over how many renewal cycles?"

Lay out your approach

Estimand. The causal effect of the price increase on revenue and churn in that country, to decide keep / roll back / scale.
Name the threat. No user-level randomization (fairness, perception, legal), so I need a credible counterfactual for "that country, absent the change."
Design: a quasi-experiment — synthetic control (or difference-in-differences) from comparable markets that didn't change price, with the rollout date as the intervention.
Validate on the pre-period — the synthetic must track the treated country before the change, or I don't trust the gap.
Read churn across renewal cycles, then net the subscriber loss against the ARM gain.

Then check in: "Since we can't randomize price, I'd build a synthetic counterfactual from other markets and read the post-period gap. Want me to walk through how the weights are actually fit?"

Simulate the back-and-forth

Interviewer

Why not just compare revenue before and after the price change in that country?

You

Because a pure before/after confounds the price change with everything else that month — a competitor move, a tentpole release, FX. I need a counterfactual for what that country would have done without the change, built from markets that didn't change price.

Interviewer

How do you build that counterfactual — just average the other markets?

You

Not a flat average — a weighted blend, synthetic control. I solve for non-negative weights that sum to one so the blend reproduces the treated country's pre-period curve with the smallest squared error. A mix like 0.5 Canada, 0.3 Australia, 0.2 Spain isn't hand-picked — it's the solution to that fit.

Interviewer

Suppose the actual line drops to an index of 93 while the synthetic sits at 101. What's the effect?

You

The gap — about minus seven on the subscriber index — is the estimated causal effect on subscribers. The validity check is that the two lines sit on top of each other before the change and only separate after. For churn I'd read across at least two renewal cycles, because people cancel at their next renewal, so a four-week read understates it.

Interviewer

With one treated country, how do you get uncertainty?

You

Placebo / permutation tests — pretend each donor market was treated and refit; the real effect should stand out against that placebo distribution. Then I net the subscriber loss against the ARM increase for the revenue answer.

Land a clear conclusion

You

So: this is causal inference, not an A/B test. Build a synthetic control from non-price-change markets, confirm the pre-period matches, read the post-period gap as the effect, get uncertainty from placebo markets, and read churn across renewal cycles before netting against ARM. The pre-period fit is what decides whether I trust the number.

Key summary — what really matters.

The estimand is the causal effect of the price change on revenue and churn — to keep / roll back / scale.
No user randomization → build a counterfactual, don't read a raw before/after.
Synthetic control: non-negative weights summing to 1, fit on the pre-period.
The effect is the post-period gap $\tau(t) = Y_{\text{treated}}(t) - \hat{Y}_{\text{treated}}(t)$ ; validate on the pre-period.
Placebo tests for uncertainty; read churn across renewal cycles, then net against ARM.

Failure mode. A pure pre/post comparison in the treated country confounds the price change with everything else that month (a competitor move, a tentpole release, FX).

The method. DiD assumes parallel trends — treated and control would have moved together absent the change; test it on the pre-period. Its estimator is the difference of the two before/after changes:

$\hat{\tau}_{\text{DiD}} = \big(\bar{Y}_{T,\text{post}} - \bar{Y}_{T,\text{pre}}\big) - \big(\bar{Y}_{C,\text{post}} - \bar{Y}_{C,\text{pre}}\big)$

Synthetic control is more defensible at the country level: you can't observe the treated country "without the price change," so you build that counterfactual as a weighted blend of donor markets, with weights chosen so the blend matches the treated country before the change. The effect is the post-period gap:

$\hat{Y}_{\text{treated}}(t) = \sum_i w_i\, Y_i(t), \quad w_i \ge 0,\ \textstyle\sum_i w_i = 1 \qquad\Longrightarrow\qquad \tau(t) = Y_{\text{treated}}(t) - \hat{Y}_{\text{treated}}(t)$

How the weights are found (the actual math). You don't pick them by hand — you solve for the blend that tracks the treated country most closely over the pre-period while staying a genuine weighted average:

$w = \arg\min_{w}\ \sum_{t<0}\Big(Y_{\text{treated}}(t) - \sum_j w_j Y_j(t)\Big)^2 \quad \text{subject to}\quad w_j \ge 0,\ \textstyle\sum_j w_j = 1$

The constraints $w_j \ge 0$ and $\sum_j w_j = 1$ keep the synthetic an interpolation of real markets (no extrapolation), which also makes the weights sparse and interpretable. A 0.5·Canada + 0.3·Australia + 0.2·Spain split is simply the solution — those markets, in that mix, reproduced the treated country's pre-period curve with the smallest squared error (Abadie–Diamond–Hainmueller).

Sample data (active-subscriber index, 100 = baseline; price change at month 0):

Month (rel.)	Treated (actual)	Synthetic control	Gap τ
−2	100	100	0
−1	101	101	0
0	100	100	0
+1	96	100	−4
+2	94	101	−7
+3	93	101	−8

Treated (actual)Synthetic control

Illustrative: the synthetic matches the treated country's pre-period, then the actual diverges after the price change. The widening gap is the estimated effect on subscribers.

Read churn across renewal cycles — members cancel at their next renewal, so a 4–8 week read understates it. With one treated unit, inference comes from placebo/permutation tests (pretend each donor is treated; the real effect should stand out). Then combine the subscriber effect with the ARM increase to get the net revenue effect.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you pin down the causal quantity, the population, and the decision before any math.	You jump to a test or a number without saying what you're estimating or why.	You name the estimand, the unit, the population, and the decision it drives.
Design validity	Whether the design actually identifies the effect — randomization unit, interference, selection/confounding.	You analyze a biased or diluted comparison as if it were clean.	You pick the unit that contains interference, name the threat and its direction, and control it.
Statistical rigor	Whether the stats hold up — power and sample size, variance, the right test, and the assumptions behind them.	You misuse a test, ignore power or variance, or lean on an unchecked assumption.	You size the test, reduce or adjust variance, choose the fitting test, and state what would break it.
Decision & trade-offs	Whether you turn the result into a call — guardrails, cost, and what would flip it.	You stop at "significant / not," or ship on one metric while a guardrail breaks.	You land a ship/hold call, weigh the trade-offs, and name the number that changes it.
Communication	Whether you reason clearly, lead the conversation, and translate the stats for a stakeholder.	You dump jargon, or can't explain the method in plain words.	You reason out loud in order, check in, and give a plain recommendation.

Interview tips

Strong-answer signal: reaching for synthetic control with placebo inference, checking the pre-period match, and reading churn across renewal cycles before netting against ARM. Common trap: a pure pre/post comparison with no control, confounding the price change with everything else that month.