Probability & Statistics

The machinery every other track leans on. Twenty-five lessons built on real data, from the US jobs report to Student's 1908 sleep study, with simulations wherever the truth needs to be known.

25 lessons in 6 modules, all running on one product's data and building on each other. Start at the top.

0 of 25 done
Start →

Module 1 · Data and Randomness

What you are actually measuring, and the rules randomness obeys.

  1. 1Population and SampleParameter versus statistic, where sampling error comes from and how fast it shrinks, and why a bigger sample can't fix a biased one. With the US jobs report's published margin of error, the 1936 Literary Digest poll, and a 250,000-person weekly survey that did no better than 10 people picked at random.
  2. 2Describing a DistributionCentre, spread and shape: mean versus median, variance and standard deviation, quantiles, and when the 68-95-99.7 rule doesn't hold. With US household income, where the mean sits 45% above the median, adult heights, where they match, and why Google's engineers measure latency in percentiles.
  3. 3Probability RulesThe basic rules, unions and intersections, independence, complements, and counting. With the 1-in-73-million figure that the Royal Statistical Society called invalid, Google's finding that memory errors cluster, and the birthday problem.

Module 2 · Conditioning and Random Variables

What one fact tells you about another, and how to put a number on a random thing.

  1. 4Conditional ProbabilityConditional probability and the law of total probability, and why P(A given B) and P(B given A) answer different questions. With US unemployment data, where college graduates are both the largest group of unemployed people and the least likely to be unemployed.
  2. 5Bayes' TheoremBayes' theorem, the base-rate trap, and stacking evidence with odds. With the mammography question that only 21% of 160 gynaecologists answered correctly, and that 87% got right once it was put in counts.
  3. 6Random VariablesPMF, PDF and CDF, expectation and variance, and the linearity rule that holds without conditions. With the Powerball prize table, where a $2 ticket is worth about 32 cents before the jackpot, and the MPG illusion, where a curved scale misleads people about fuel savings.
  4. 7Joint Distributions and CorrelationJoint, marginal and conditional distributions, covariance, correlation and R-squared, and why uncorrelated isn't independent. With the Titanic inquiry's survival figures by class, and Anscombe's four datasets that share every summary statistic and look nothing alike.

Module 3 · The Distributions

The handful of shapes real data keeps turning out to have.

  1. 8Discrete DistributionsBernoulli, binomial, geometric, negative binomial and Poisson, and the variance-to-mean check that tells them apart. With the 1946 analysis of flying bombs on south London, a near-perfect Poisson, and what changes when every user has their own rate.
  2. 9Continuous DistributionsUniform, exponential and normal, standardising and z-scores, and the memorylessness that makes the exponential the wrong model for most churn. With US adult heights, which match the normal curve almost percentile for percentile.
  3. 10The Normal Family: Chi-Square, t and FWhere the chi-square, t and F distributions come from and why each one exists. With the 1908 paper by 'Student' that introduced the t distribution, worked on ten patients and two sleeping drugs, and what a '95%' interval built with 1.96 really covers on a small sample.
  4. 11Choosing a DistributionHow the families connect, how to pick one, heavy tails and the lognormal. With US household income, where a lognormal fitted from just the median and the mean lands within 2% of the published percentiles from the 40th up, and misses badly at the bottom.

Module 4 · From Sample to Population

Why a number computed from 500 people says anything about five million.

  1. 12Sampling Distributions and Standard ErrorThe sampling distribution of a statistic, and the difference between a standard deviation and a standard error. With a national health survey that reports both for US men's height, and whose standard error is nearly twice what the textbook formula gives.
  2. 13The Central Limit TheoremWhat the central limit theorem promises, how large n has to be before it delivers on a skewed metric, and where it fails entirely. On a population shaped like US household income, a '95%' interval at n = 30 covers 90%, and every miss is on the same side.
  3. 14Estimators and Maximum LikelihoodBias, variance, consistency and mean squared error, then maximum likelihood. With the German tank problem, where Allied economists estimated tank production from captured serial numbers and came within a few dozen tanks of Germany's own records.
  4. 15The BootstrapResampling your own data to get a standard error for a statistic that has no formula, percentile intervals, and the cases where the method breaks. Worked on the ten patients from Student's 1908 sleep study, where it handles the median easily and can't say anything sensible about the maximum.

Module 5 · Inference

Turning a sample into a claim, and knowing how often the claim is wrong.

  1. 16Confidence IntervalsWhat 95% confidence actually means, shown by drawing intervals until the share that contain the truth settles, how the US jobs report words its own interval, and why the proportion interval everyone learns covers far less than it claims at product-sized rates.
  2. 17Hypothesis Testing and P-ValuesNull and alternative, test statistic, rejection region, and what a p-value is and isn't. With Fisher's tea-tasting test, the American Statistical Association's six principles on p-values, and a project that re-ran 100 psychology studies and found 36% significant again.
  3. 18Type I, Type II and PowerThe two kinds of error, the two overlapping distributions and the areas under them, and how sample size, effect size and alpha move those areas. At a sample size teams often use, a real 15% lift is missed two times in three, and a review of neuroscience found the same problem across a whole field.
  4. 19Comparing Two GroupsTwo means with the pooled and Welch t-tests, two proportions, and paired data. With Student's sleep study, where the same ten patients give p = 0.003 when analysed as pairs and p = 0.08 when they aren't, and a simulation where the pooled t-test fires half the time on nothing.
  5. 20Chi-Square Tests for Categorical DataGoodness of fit, independence and homogeneity, plus Fisher's exact test and McNemar. With Mendel's pea counts, the Titanic's survival table by class, and an exact calculation showing that plain chi-square holds up on small tables while the usual corrections throw away power.
  6. 21Nonparametric and Permutation TestsPermutation tests and Mann-Whitney, what each one actually tests, and how a rank test and a t-test can both be right and still disagree. With an exact permutation test on Student's ten patients, and a medical statistician's warning that rank tests in large studies answer the wrong question.
  7. 22Bayesian InferencePrior to posterior, the beta-binomial update, and a credible interval versus a confidence interval. With the search for Air France 447, found within a week once a Bayesian analysis guided it, and an A/B test where p = 0.16 and a 92% chance that treatment is better are both true.

Module 6 · The Statistics Behind an A/B Test

The same machinery, in the place you will be asked about it.

  1. 23An Experiment as a Two-Sample ProblemThe sampling distribution of a difference, and where an experiment's standard error comes from. Measuring a difference takes four times the traffic of measuring a level, which is why Microsoft's experimentation team built a method that gets the same power from half the users.
  2. 24Choosing a Test for an ExperimentMetric shape to test, the central limit theorem argument that allows a t-test on revenue, and where that argument runs out. The name of the test barely matters; analysing the wrong unit can triple the false positive rate.
  3. 25Alpha, Power and Sample SizeAlpha, power, the minimum detectable effect and sample size as four dials that trade against each other, and why the MDE is set by the business rather than by statistics. With Microsoft's Bing, where a 1% revenue change is worth over $10 million a year, and the hundredfold traffic cost of detecting it.