Crash Courses

Master the concepts behind the answers. Each track runs in order, from the idea it is built on to the version you would actually defend in an interview — start at the top, or pick out the one method you need. Concept links in the practice answers bring you here.

Advanced Experimentation

10 courses

0 of 10 done

Bayesian A/B Testing

Prior to posterior on a conversion rate, the beta-binomial worked end to end, and P(B beats A): the question VWO's statistician says users ask most. The posterior is honest at every moment, but checking it daily and stopping at 95% ships a winner in about 18% of tests where nothing changed.

Expected Loss and the Decision Rule

Turning a posterior into a decision: expected loss, the region of practical equivalence, and why a probability of winning isn't a reason to ship. With VWO's 'threshold of caring' and Microsoft's finding that only about a third of ideas improve the metric they target.

Multi-Armed Bandits and Thompson Sampling

Thompson sampling, the trade a bandit actually makes, and the two conditions it needs. With Thompson's 1933 idea, Yahoo's evidence that it's 'highly competitive', and a proof that adaptively collected data underestimates the arms it stops sampling.

Stratified and Blocked Randomisation

Balancing covariates by construction rather than by luck, plus blocking and re-randomisation. What stratifying can buy is capped by one number, the covariate's correlation with the outcome, and a World Bank study found the gains matter most in small samples.

Cluster Randomisation and the Design Effect

Randomising groups instead of users: the intra-cluster correlation, the design effect, and what your effective sample size really is. Why LinkedIn and Facebook built cluster-randomised systems, and why an ICC of just 0.01 can make a user-level analysis fire on nothing 72% of the time.

Switchback Experiments

Turning the whole system on and off over time when a change touches a shared marketplace. Why Lyft showed a user-level A/B can be six times too big, how DoorDash randomises 30-minute windows by city, and the three traps: time of day, carryover, and the unit of analysis.

Interleaving for Ranking Evaluation

Blending two rankings into one list so each user compares them directly. Netflix found it needed more than 100 times fewer members than its most sensitive A/B metric; here's where that gain comes from, why it depends on your data, and what interleaving can't tell you.

Long-Term Holdouts and Effect Decay

Measuring what a two-week test can't see, using a holdout you keep running. With Google's research on ads blindness, which showed short-term results can mislead about the long term, and led Google to cut the ad load on mobile search by half.

Meta-Analysis Across Experiments

Learning from all your past experiments rather than one: the distribution of true effects, the winner's curse, and shrinkage. Facebook cut the error in 226 News Feed test results by 44% by shrinking them, and Bing's experiments turned out to have very fat tails.

Building an Experimentation Platform

Metric governance, the guardrail checks, and the gates between a readout and a launch. Why careful experiments still produce hundreds of false wins a year at scale, and what Booking.com and Microsoft built to stop them: real-time monitoring, pre-registration, and checks that hide results when the data is broken.

Causal Inference

27 courses

0 of 27 done

Potential Outcomes and the Counterfactual

Every causal question compares what happened with what would have happened otherwise, and you never see both for the same person. How to write that down, the three different averages people mean by 'the effect', and why randomisation lets you estimate them anyway. With GitHub's randomised experiment on Copilot, where developers with the AI assistant finished a coding task 55.8% faster.

Causal Impact (Counterfactual Forecasting)

Estimating the effect of a change you couldn't randomise by forecasting what the metric would have done anyway, then reading the gap. How Google's CausalImpact recovered an ad experiment's answer using only search trends, and why streaky daily noise makes a naive test cry wolf.

Selection Bias

Comparing people who got a treatment with people who didn't measures the treatment plus how different those people already were. How to split a naive comparison into the real effect and the selection bias, where selection comes from, and how to predict its direction. With Facebook's ads, where selection made the effect look four times bigger, and hormone therapy, where it reversed the sign.

Causal Graphs: Confounders, Mediators, Colliders

A causal graph draws your assumptions about what causes what, and tells you which variables to control for. The three structures every graph is made of, why controlling for the wrong one adds bias instead of removing it, and why filtering your data is a form of control. With Facebook's ads, Google's ads blindness, and the birth weight paradox.

The Back-Door Criterion

Which variables do you need to control for to get a causal effect from observational data? The back-door criterion answers that from the causal graph. How it works, why adding more controls doesn't always help, and why the one confounder that matters can be the one you didn't measure. With Facebook's study, where adding census data barely moved a biased estimate and adding activity during the campaign nearly fixed it.

Media Mix Modeling (MMM)

Estimating what each marketing channel contributes to sales from aggregate spend and sales data, with carryover and diminishing returns built in. Why Google's own researchers call its results correlational, why the budget decision runs on marginal return, and how experiments keep the model honest.

Heterogeneous Treatment Effects (CATE)

An average effect can hide a change that helps some users and hurts others. How to find who it helps without fooling yourself: the aspirin trial that 'failed' for Geminis, honest estimation, causal forests and meta-learners, and how Uber targets promotions on effect minus cost.

Regression Adjustment

The most common way to 'control for' differences between treated and untreated users is a regression with the treatment and some covariates. What that coefficient means, what it rests on, and why adding controls can move the answer away from the truth as easily as toward it. With Facebook's study, where regression gave estimates from 99% down to 37%, and in one campaign the wrong sign, against known experimental answers.

Incrementality Testing (Ghost Ads, PSA & Geo Lift)

Measuring the sales an ad actually caused, not the ones that would have happened anyway. How eBay found its search ads lost money when regression said they returned 4,100%, how ghost ads and geo experiments get the real number, and why these tests need millions of users.

Propensity Scores

A propensity score is each user's probability of being treated, given what you know about them. Comparing users with the same score adjusts for all those characteristics at once. How to estimate one, how to use it, how to check overlap and balance, and what it can't fix. With Facebook's study, where matching balanced the variables in the model but left others a full standard deviation apart.

Cold-Start Markets

Every causal method you know needs history for the treated unit, and a brand-new city has none. Start with the design that avoids the problem, a randomised rollout order that public health has used since the Gambia hepatitis trial, then benchmark against past launches and pool early weeks instead of reacting to them.

Matching and Inverse-Probability Weighting

Two ways to turn a propensity score into a fair comparison: pair each treated user with a similar untreated one, or reweight everyone so the groups look alike. How each works, step by step on a small example, what goes wrong when weights get extreme, and why a perfect balance table doesn't mean a right answer. With Facebook's study, where both methods still overstated the ad effect by close to double.

Doubly Robust Estimation

A doubly robust estimator combines a model of the outcome with a model of who gets treated, and is right if either one is right. How it works, why it's the default in modern causal toolkits, and what it doesn't protect you from. With Facebook's study, where the doubly robust method did no better than the others, because the problem was a confounder neither model had.

Before-and-After and Interrupted Time Series

When a change ships to everyone, the obvious measure is the metric before against after. Why that usually measures the calendar, how an interrupted time series does better by modelling the trend and the weekly cycle, and what it still can't handle. Worked on real Wikipedia traffic around a 'launch' that never happened.

Difference-in-Differences

When one group gets a change and another doesn't, compare how much each one changed. Subtracting the comparison group's change removes everything that hit both groups over the same period. How difference-in-differences works, what it assumes, and how to check it. With the classic study of New Jersey's 1992 minimum wage rise, and Wikipedia traffic showing what a year-earlier comparison removes.

The Parallel Trends Assumption

Difference-in-differences only works if the treated and comparison groups would have moved together without the treatment. You can never check that directly. How to look for evidence with an event study, why passing a pre-trend test proves less than it seems, and what to do instead. With the study of Facebook's arrival at US colleges, and Roth's work on how weak pre-trend tests are.

Staggered Rollouts

Products often launch in waves: some countries this month, others next quarter. The standard regression for this, two-way fixed effects, can badly mis-measure the effect when it changes over time, and can even get the sign wrong. Why that happens, and the newer estimators built to fix it. With Facebook's college-by-college rollout as a real example.

Fixed Effects

A fixed effect gives each unit, or each period, its own baseline, so the comparison happens within a unit rather than across units. That removes every stable difference between units, measured or not. How it works, what it can't remove, and how the study of Facebook at US colleges used it.

Synthetic Control

When one city, one country or one state gets a change, no single comparison unit looks enough like it. Synthetic control builds one: a weighted mix of untreated units that tracks the treated unit closely before the change. How it works, why the weights are kept positive and summing to one, and when it fails. With California's 1988 tobacco programme and German reunification.

Inference for Synthetic Control

With one treated unit, the usual standard errors don't apply. How to judge whether a synthetic control gap is real: run the same method on every untreated unit and see where the treated one ranks, move the treatment date back in time, and drop donors one at a time. With the checks behind California's tobacco study and German reunification.

Counterfactual Forecasting

When a change hits everything at once and there's no comparison group, you can still forecast what would have happened from the metric's own history and related series the change didn't touch. How Google's CausalImpact does it, how well it matched a randomised experiment, and how to measure your method's noise floor before trusting it. With real Wikipedia traffic as the backtest.

Geo Experiments and Incrementality

Ad platforms report the sales that followed an ad. What you need is the sales the ad caused. A geo experiment measures that by turning advertising on or off in randomly chosen regions. How Google designs them, how to analyse them, and what they've found: from paid clicks that were 80% incremental to brand ads that were almost entirely wasted.

Instrumental Variables

Instrumental variables and two-stage least squares, what makes an instrument valid, whose effect you are estimating, and what a weak instrument costs. With the Vietnam draft lottery, eBay's paid search experiment, and a simulation of 200 studies that share one weak instrument.

Encouragement Designs and LATE

When you can't make anyone use a feature, randomise the invitation instead. Intention to treat, the effect on compliers, why comparing users with non-users fails even inside an experiment, and what a weak nudge costs. With Microsoft's and Accenture's GitHub Copilot experiments and the Coronary Drug Project.

Regression Discontinuity

Using a threshold somebody else already set, choosing a bandwidth, and the assumption that actually gets violated in practice. With Yelp's half-star rounding, which let researchers measure what an extra star does to a restaurant's revenue and its chance of selling out.

Heterogeneous Treatment Effects (CATE)

S-, T- and X-learners for estimating who the effect is biggest for, validating a model of something you never observe, and what a targeting rule is worth. With a voter-turnout letter re-analysed by the X-learner's authors, and a causal forest whose varied estimates turned out to be mostly noise.

Sensitivity Analysis and Choosing a Method

How strong an unmeasured confounder would have to be to overturn a result, and a decision procedure for choosing among the methods in this track. With the smoking and lung cancer debate, hormone therapy, a survey of Darfur refugees, and California's tobacco programme, whose parts no method could separate.

Concepts

6 courses

0 of 6 done

Surrogate Metrics & the OEC

Deciding on a long-term outcome you can't wait for, by measuring an early signal that stands in for it. How Netflix checked a two-week surrogate against two-month results in 200 A/B tests, and how a heart drug that fixed its surrogate killed more patients.

Customer Lifetime Value (LTV)

How much a customer is worth over the whole relationship, and why the textbook shortcut gets it wrong. Why churn falls over time even when no customer changes, and how a simple model projected 12 years of real subscription retention from 7 years of data, within a few percent.

Price Elasticity of Demand

How much demand moves when the price changes, and when a price rise raises revenue. How Uber measured elasticities of about −0.5 from the rounding in its surge prices, and why Netflix's 2019 US price rise can't be read from a before-and-after.

Prevalence Estimation (Design-Based Sampling)

How platforms measure what share of views went to harmful content, when the harm is rare and every label is expensive. Meta's and YouTube's published prevalence metrics, and how Pinterest samples where the violations are and still gets an unbiased number.

Label-Error Correction (Rogan–Gladen)

An imperfect labeller biases a prevalence estimate, and when the thing you're counting is rare, a tiny false-positive rate can double it. The one-line correction from 1978 epidemiology, how it decided a famous COVID antibody study, and how Pinterest decides when to use it.

LLM-as-Judge for Measurement

Using a large language model as the labeller behind a metric, and the checks that make its labels trustworthy. How OpenAI and Pinterest run the golden-set loop and launch gate, the biases LLM judges are known to have, and when a panel of judges beats one.

Experimentation

20 courses

0 of 20 done

Randomisation and What an Experiment Proves

Why companies like Microsoft test ideas instead of trusting judgement, why before-and-after comparisons and comparing users who chose a feature both mislead, and what random assignment fixes. With Facebook's ad experiments, where comparing exposed users said +316% and the randomised test said +73%.

Pre-Registration and the Design Doc

Why the analysis has to be decided before the data arrives, what a design document contains, and how Microsoft's experimentation team checks a test before it starts. With the study where 29 teams analysed the same data and got answers ranging from slightly fewer red cards to nearly three times as many.

Choosing the Randomisation Unit

Users, devices, sessions, teams or regions: what gets randomised changes what you can measure, how much traffic you need, and whether the comparison is clean. How assignment actually works, and why analysing a smaller unit than you randomised makes false positives far more common.

The OEC and Guardrail Metrics

An experiment needs one metric that decides success and a set of metrics that must not get worse. How to choose them, with the Bing bug that made search results worse and raised revenue per user by 30%, and how Airbnb's guardrail system decides which experiments need a second look.

Power and Sample Size

How many users an experiment needs, and why. The four quantities that set sample size, why halving the effect you want to detect quadruples the users you need, and what goes wrong when a test is too small. With Bing's slowdown experiment, where every 100 milliseconds cost 0.6% of revenue.

A/A Tests and Platform Validation

An A/A test gives both groups the same product, so any 'effect' it finds is a bug or chance. How to use A/A tests to check an experimentation platform, why the whole spread of p-values matters more than one result, and what Microsoft's A/A tests have caught: bots, a bucket assignment bug, and effects that carried over for three months.

Sample Ratio Mismatch

When the groups in an experiment aren't the sizes you configured, the result can't be trusted, and it can even point the wrong way. How to check for a sample ratio mismatch, why about 6% of Microsoft's experiments have one, and where they come from. With the MSN test that looked like a loss until bot filtering was found to be removing the treatment's most engaged users.

Experiment Data Quality

A logging problem that hits both groups equally costs you some precision. One that hits one group more than the other can invent a result. How to tell the two apart, why the change being tested is so often what breaks the logging, and the checks to run before reading any metric. With Bing's experiments where a slower page looked like it got more clicks.

Reading a Result: P-Values and Confidence Intervals

What a confidence interval and a p-value actually say, what they don't, and why a significant result is wrong more often than 5% of the time. With the American Statistical Association's principles on p-values, and Kohavi, Deng and Vermeer's estimates showing that at companies where about 10% of ideas work, about 22% of significant wins are false.

Choosing the Test

Which test to use for a conversion rate, an average, a ratio or a skewed revenue metric, and which common warnings you can ignore. Why the t-test works on data that isn't normal, why '30 users is enough' is far too few for revenue, and why two metrics both called 'click-through rate' can give different answers.

Ratio Metrics and the Delta Method

Click-through rate per page view, revenue per order and queries per session are ratios whose denominator isn't the randomised user, and the obvious standard error for them is wrong. How the delta method fixes it using only per-user totals, and what happens to the false positive rate if you skip it.

Heavy Tails and Capping

When a few users account for most of a metric, its variance explodes and experiments need huge samples. How capping fixes that, what it changes about the metric, and how to choose a cap honestly. With Bing's revenue per user, where a $10-a-week cap cut skewness from 18 to 5.3 and let the same test detect changes 30% smaller.

CUPED and Variance Reduction

Much of the noise in an experiment comes from users simply being different from each other, and you already know how they behaved before the test. CUPED uses that to cut the noise. How it works, why it doesn't bias the result, and what it bought Bing: about half the variance, the same as doubling traffic.

Sequential Testing and Peeking

Checking an experiment every day and stopping as soon as it looks significant makes false positives several times more common. Why that happens, how sequential methods let you look early without the cost, and why stopping a harmful test early is fine. With the always-valid p-values developed at Optimizely, and Microsoft's alerting.

Multiple Testing: FWER and FDR

An experiment that tracks 50 metrics will almost always show a 'significant' change in one of them, even when nothing happened. Why, what you can control (the chance of any false win, or the share of wins that are false), and how to set up metrics and segments so the readout means something. With Airbnb's calculation that 50 guardrails at 5% give a 92% chance of a false alert.

Novelty and Primacy Effects

An effect measured in the first weeks of an experiment isn't always the effect you'll get after launch. Users try new things out of curiosity, struggle with changes before adapting, and slowly learn to ignore what doesn't help them. How to tell these apart from noise, with MSN's Hotmail link, Microsoft's 'trends' that came from an A/A test, and Google's study of ads blindness.

Interference and SUTVA

A standard A/B test assumes each user's outcome depends only on their own group. In social products and marketplaces that's often false: effects spread between friends, and groups compete for the same drivers or listings. How interference biases results, and how LinkedIn, DoorDash and Airbnb design experiments around it.

Triggered Analysis and Dilution

When a change only affects users who reach a certain screen or situation, analysing everyone dilutes the effect and wastes traffic. How to analyse only the users who could be affected, why that needs the same trigger logged in control, and how to report the result. With Microsoft's guidance on counterfactual logging and Bing's malware-ad experiment on 3.8 million triggered users.

Segment Effects and Interaction Tests

An average effect can hide a change that helps some users and does nothing, or harm, for others. How to look for differences between segments without fooling yourself, why 'significant in one segment but not the other' isn't evidence of a difference, and when a segment result is worth acting on. With eBay's search ads, which worked for new and occasional buyers and not at all for the frequent ones who drew most of the spend.

Shipping the Decision

Everything in this track as one procedure for deciding whether to ship: check the experiment is valid, read the primary metric against what matters, ask what the test could have detected, check the guardrails, ask whether the result is believable, and look for who it helps and how it lasts. Each step illustrated with a real decision at Bing, MSN or elsewhere.

Product Analytics

21 courses

0 of 21 done

The Common Metrics Every Product Runs On

A general metrics framework that works for any product: active users and stickiness, a North Star that fits the type of business, then the user journey from acquisition to revenue, with the numbers real companies actually report.

Define the Metric Before You Move It

Meta, Snap, Reddit, Roblox and Duolingo all report daily active users, and they all mean something different by it. What a metric definition has to pin down, and how to choose one that fits the decision.

Which Metric Should This Business Care About

Start from how the business makes money, find the user action that creates that money, and test the result. A step by step way to find a business's North Star, with the numbers real companies actually report.

The Growth Equation

One equation ties acquisition, retention, churn and resurrection together. Learn each part, see how Duolingo used it to grow daily users 4x, and play with it to see why retention sets the ceiling on growth.

Distributions: Why User Data Lies

On a typical day, only about 1 in 22 Playtika players spends any money. Why user data is almost always lopsided, why the average can describe almost nobody, and which summary answers which question.

Acquisition: Where Users Come From

What acquisition is, why it matters, the channels users come through, and the metrics that tell a good channel from an expensive one. With what Airbnb learned when it paused paid marketing, and what Google pays to be the default search engine.

Activation: The First Experience

Activation is the moment a new user first gets real value from the product. Why it predicts who stays, how Slack found its threshold of 2,000 messages, and a step by step way to find and improve your own.

Retention Curves

Retention asks how many of the people who started are still here. Learn the three standard ways to define it, how to read the shape of a retention curve, how to read a cohort table, and the mistake that makes recent cohorts look worse than they are.

Churn: Defining It Before Predicting It

Churn is the users you lost. For a subscription, that's easy to see. For most apps, nobody tells you they've left. How to define churn, pick an inactivity threshold, turn monthly churn into yearly, and measure churn in dollars.

Resurrection and Win-Back

Resurrected users are people who left and came back. In a mature product, the pool of past users keeps growing, so this quiet part of the growth equation can get big. How to define it, measure it, and tell whether a win-back campaign really worked.

Funnel Analysis and Drop-off

A funnel follows users through the steps toward a goal and shows where they drop off. How to read step and overall conversion, which step to fix, and how to split a change in conversion exactly across steps.

Additive and Multiplicative Decomposition

Most metrics are either a sum of parts or a product of drivers. How to split a change in each kind, why the obvious split leaves a leftover term, and how logs split a product exactly. Worked on Reddit, Uber and Meta's reported numbers.

Mix Shift and Simpson's Paradox

A blended average can move even when no group inside it changes, just because the groups changed size. How mix shift works, how to split a change into what happened within groups and what happened to the mix, and when it flips the answer entirely.

Segmentation: Choosing the Cut

The same number can be split by country, platform, plan, channel or a dozen other dimensions, and each split tells a different story. How to pick the cut from what could have changed, why slicing everything always finds something, and what makes a cut worth running.

Forecasting and Baselines

To say a number fell, you need to know what it should have been. How to build a baseline that respects the weekly cycle, score it honestly, and put a range around it, worked on 63 days of real Wikipedia traffic.

Sizing the Opportunity

A finding becomes a decision once it has a number on it. How to size a problem as volume times gap times value, why the gap has to be measured rather than assumed, and why the size of a problem isn't the same as what you can win back. Worked on Google's speed experiment and Netflix's password sharing.

Data Quality: Doubt the Instrument First

A number can change because users changed, or because the way you measure them changed. How to tell the two apart, the checks that catch a broken pipeline even when nothing looks wrong, and why the worst errors make metrics look better, not worse. Worked on Bing's click tracking and Twitter's three-year user overcount.

Why Did It Move?

One procedure for every 'why did this metric change' question, run in the same order each time: is it real, is the data right, which part of the equation moved, which cut shows it, what's the size, what do we do. Worked on Netflix's subscriber loss in early 2022.

Attribution Models

A customer saw three ads before buying. Which one gets the credit? How the standard attribution models split it, why they all measure which channels were present rather than which ones made a difference, and what eBay and Facebook found when they tested attribution against experiments.

Monetization: LTV, Payback and the Formula That Lies

Revenue breaks into users, the share who pay, and what each payer brings in. How to estimate what a user is worth over time, why the textbook LTV formula gets it wrong for most products, and why payback period is the number that decides spending. Worked on Spotify and Snowflake.

Growth Loops and Virality

A funnel ends. A loop feeds itself: new users bring in more new users. How to measure a loop with the viral coefficient, why a coefficient well below 1 is still valuable, why cycle time matters as much, and what Zoom and Dropbox's filings say about loops built into the product.

Statistics

27 courses

0 of 27 done

Population and Sample

Parameter versus statistic, where sampling error comes from and how fast it shrinks, and why a bigger sample can't fix a biased one. With the US jobs report's published margin of error, the 1936 Literary Digest poll, and a 250,000-person weekly survey that did no better than 10 people picked at random.

Entity Resolution & Probabilistic Record Linkage

Decide which identifiers belong to the same person — deterministic vs probabilistic matching, match scores, the base-rate trap, and turning pairwise links into clean clusters.

Describing a Distribution

Centre, spread and shape: mean versus median, variance and standard deviation, quantiles, and when the 68-95-99.7 rule doesn't hold. With US household income, where the mean sits 45% above the median, adult heights, where they match, and why Google's engineers measure latency in percentiles.

Probability Rules

The basic rules, unions and intersections, independence, complements, and counting. With the 1-in-73-million figure that the Royal Statistical Society called invalid, Google's finding that memory errors cluster, and the birthday problem.

Conditional Probability

Conditional probability and the law of total probability, and why P(A given B) and P(B given A) answer different questions. With US unemployment data, where college graduates are both the largest group of unemployed people and the least likely to be unemployed.

Bayes' Theorem

Bayes' theorem, the base-rate trap, and stacking evidence with odds. With the mammography question that only 21% of 160 gynaecologists answered correctly, and that 87% got right once it was put in counts.

Random Variables

PMF, PDF and CDF, expectation and variance, and the linearity rule that holds without conditions. With the Powerball prize table, where a $2 ticket is worth about 32 cents before the jackpot, and the MPG illusion, where a curved scale misleads people about fuel savings.

Joint Distributions and Correlation

Joint, marginal and conditional distributions, covariance, correlation and R-squared, and why uncorrelated isn't independent. With the Titanic inquiry's survival figures by class, and Anscombe's four datasets that share every summary statistic and look nothing alike.

Discrete Distributions

Bernoulli, binomial, geometric, negative binomial and Poisson, and the variance-to-mean check that tells them apart. With the 1946 analysis of flying bombs on south London, a near-perfect Poisson, and what changes when every user has their own rate.

Continuous Distributions

Uniform, exponential and normal, standardising and z-scores, and the memorylessness that makes the exponential the wrong model for most churn. With US adult heights, which match the normal curve almost percentile for percentile.

The Normal Family: Chi-Square, t and F

Where the chi-square, t and F distributions come from and why each one exists. With the 1908 paper by 'Student' that introduced the t distribution, worked on ten patients and two sleeping drugs, and what a '95%' interval built with 1.96 really covers on a small sample.

Choosing a Distribution

How the families connect, how to pick one, heavy tails and the lognormal. With US household income, where a lognormal fitted from just the median and the mean lands within 2% of the published percentiles from the 40th up, and misses badly at the bottom.

Sampling Distributions and Standard Error

The sampling distribution of a statistic, and the difference between a standard deviation and a standard error. With a national health survey that reports both for US men's height, and whose standard error is nearly twice what the textbook formula gives.

The Central Limit Theorem

What the central limit theorem promises, how large n has to be before it delivers on a skewed metric, and where it fails entirely. On a population shaped like US household income, a '95%' interval at n = 30 covers 90%, and every miss is on the same side.

Estimators and Maximum Likelihood

Bias, variance, consistency and mean squared error, then maximum likelihood. With the German tank problem, where Allied economists estimated tank production from captured serial numbers and came within a few dozen tanks of Germany's own records.

The Bootstrap

Resampling your own data to get a standard error for a statistic that has no formula, percentile intervals, and the cases where the method breaks. Worked on the ten patients from Student's 1908 sleep study, where it handles the median easily and can't say anything sensible about the maximum.

Confidence Intervals

What 95% confidence actually means, shown by drawing intervals until the share that contain the truth settles, how the US jobs report words its own interval, and why the proportion interval everyone learns covers far less than it claims at product-sized rates.

Hypothesis Testing and P-Values

Null and alternative, test statistic, rejection region, and what a p-value is and isn't. With Fisher's tea-tasting test, the American Statistical Association's six principles on p-values, and a project that re-ran 100 psychology studies and found 36% significant again.

Type I, Type II and Power

The two kinds of error, the two overlapping distributions and the areas under them, and how sample size, effect size and alpha move those areas. At a sample size teams often use, a real 15% lift is missed two times in three, and a review of neuroscience found the same problem across a whole field.

Comparing Two Groups

Two means with the pooled and Welch t-tests, two proportions, and paired data. With Student's sleep study, where the same ten patients give p = 0.003 when analysed as pairs and p = 0.08 when they aren't, and a simulation where the pooled t-test fires half the time on nothing.

Chi-Square Tests for Categorical Data

Goodness of fit, independence and homogeneity, plus Fisher's exact test and McNemar. With Mendel's pea counts, the Titanic's survival table by class, and an exact calculation showing that plain chi-square holds up on small tables while the usual corrections throw away power.

Nonparametric and Permutation Tests

Permutation tests and Mann-Whitney, what each one actually tests, and how a rank test and a t-test can both be right and still disagree. With an exact permutation test on Student's ten patients, and a medical statistician's warning that rank tests in large studies answer the wrong question.

Bayesian Inference

Prior to posterior, the beta-binomial update, and a credible interval versus a confidence interval. With the search for Air France 447, found within a week once a Bayesian analysis guided it, and an A/B test where p = 0.16 and a 92% chance that treatment is better are both true.

An Experiment as a Two-Sample Problem

The sampling distribution of a difference, and where an experiment's standard error comes from. Measuring a difference takes four times the traffic of measuring a level, which is why Microsoft's experimentation team built a method that gets the same power from half the users.

Choosing a Test for an Experiment

Metric shape to test, the central limit theorem argument that allows a t-test on revenue, and where that argument runs out. The name of the test barely matters; analysing the wrong unit can triple the false positive rate.

Alpha, Power and Sample Size

Alpha, power, the minimum detectable effect and sample size as four dials that trade against each other, and why the MDE is set by the business rather than by statistics. With Microsoft's Bing, where a 1% revenue change is worth over $10 million a year, and the hundredfold traffic cost of detecting it.

Probability & Statistics: The One-Page Recap

Every formula, distribution and decision rule from the twenty-five lessons, on one page, with the real examples and measured numbers that make each one concrete. Built to be read the night before an interview.