Crash Courses
Master the concepts behind the answers. Each crash course explains one method in plain language — the intuition, the math made simple, real big-tech examples, and when to use it. Concept links in the practice answers bring you here.
Causal Inference
Synthetic Control
Build a data-driven twin of a treated market out of a blend of untreated ones, so you can measure impact even when you can't randomize.
Master it →Causal Impact (Counterfactual Forecasting)
Estimate the effect of a change you couldn't randomize by forecasting what the metric would have done anyway, then reading the gap.
Master it →Difference-in-Differences
No control group, but a change hit some units and not others at a known time? Difference out the trend and read the gap that opens up.
Master it →Media Mix Modeling (MMM)
Estimate what each marketing channel actually contributes to sales, with carryover and diminishing returns baked in, and calibrate it with experiments.
Master it →Heterogeneous Treatment Effects (CATE)
A flat average effect can hide a change that helps some users and hurts others. Estimate who it helps, then target them.
Master it →Incrementality Testing (Ghost Ads, PSA & Geo Lift)
Measure the conversions an ad actually caused — not the ones that would have happened anyway — using ghost-ad/PSA holdouts and geo lift when you can't cleanly A/B.
Master it →Experimentation
CUPED (Variance Reduction)
Use data from before the experiment even started to strip out noise you could already predict, so your test needs far fewer users to detect the same effect.
Master it →Triggered Analysis & Dilution
When a feature only fires on a slice of traffic, measuring everyone dilutes the effect toward zero. Trigger-based analysis with counterfactual logging recovers it.
Master it →Interleaving (Ranking Evaluation)
A far more sensitive way to compare two rankers than A/B testing: blend both rankings into one list and see whose results actually get clicked.
Master it →Sequential Testing & Always-Valid p-values
Why checking an experiment's results every day is riskier than it looks, and how a smarter kind of p-value lets you check anytime without fooling yourself.
Master it →Sample Ratio Mismatch (SRM)
Your A/B test was meant to be a 50/50 split but came out 50.2/49.8. Is that just luck, or a bug that makes the whole result untrustworthy? SRM is the check that tells you.
Master it →Interference & Cluster Randomization
When one user's treatment leaks onto another, ordinary A/B tests break. Here's why, and how randomizing whole clusters fixes it.
Master it →Switchback Experiments
When a change touches a shared marketplace, you can't split users. Flip the whole system on and off over time and compare the windows.
Master it →Metrics
Surrogate Metrics & the OEC
Pick the one metric you'll decide by before the test starts, and a validated early signal that stands in for a long-term outcome you can't wait months to observe.
Master it →Prevalence Estimation (Design-Based Sampling)
How to estimate how much of a rare harm is on a platform, using probability sampling and an unbiased estimator instead of a convenience sample or a model threshold.
Master it →LLM-as-Judge for Measurement
Use a large language model as a scalable labeler for measurement, with the golden-set governance and drift monitoring that make its labels trustworthy.
Master it →Label-Error Correction (Rogan–Gladen)
An imperfect labeler biases a prevalence estimate, and at low base rates a small false-positive rate can dominate. Here's the one-line correction and how to use it.
Master it →Monetization
Price Elasticity of Demand
How much quantity moves when you change price, and why 'raise the price, raise the revenue' is often wrong.
Master it →Customer Lifetime Value (LTV)
How much a customer is worth over their whole relationship with a product, not just today, and the number that decides how much you can spend to acquire them.
Master it →Statistics
Multiple Testing & the False Discovery Rate
Check enough metrics and segments and something 'significant' appears by pure chance. Here's how to control it and not fool yourself.
Master it →Entity Resolution & Probabilistic Record Linkage
Decide which identifiers belong to the same person — deterministic vs probabilistic matching, match scores, the base-rate trap, and turning pairwise links into clean clusters.
Master it →