Probability & Statistics
The machinery every other track leans on. Twenty-five lessons on one product's data, where the true parameter is known and every procedure is scored against it.
25 lessons in 6 modules, all running on one product's data and building on each other. Start at the top.
0 of 25 done
Module 1 · Data and Randomness
What you are actually measuring, and the rules randomness obeys.
- 1Population and SampleParameter versus statistic, and where sampling error comes from. Pantry's average order value is computed from every order in the week, by a query with no bug in it, and still misses the true number by 24 cents.
- 2Describing a DistributionCentre, spread and shape: mean versus median, variance and standard deviation, quantiles, and when the 68-95-99.7 rule does not hold. On Pantry's order values it covers 89.3%, not 68%.
- 3Probability RulesThe axioms, unions and intersections, independence, complements, and counting. Worked on a rare checkout bug that still has a 91.9% chance of hitting somebody in any given week.
Module 2 · Conditioning and Random Variables
What one fact tells you about another, and how to put a number on a random thing.
- 4Conditional ProbabilityConditional probability and the law of total probability, and why P(A given B) and P(B given A) answer different questions. Two such numbers about the same 337 Pantry users differ by a factor of 63.
- 5Bayes' TheoremBayes' theorem, the base-rate trap, and stacking evidence with odds. A filter that catches 94% of fraud and clears 97% of honest accounts is still wrong about 91% of the accounts it flags.
- 6Random VariablesPMF, PDF and CDF, expectation and variance, and the linearity rule that holds without conditions. Expectation always adds; variance only adds when the parts are independent.
- 7Joint Distributions and CorrelationJoint, marginal and conditional distributions, covariance, correlation and R-squared, and why uncorrelated is not independent. Two Pantry columns that determine each other exactly have a correlation of 0.001.
Module 3 · The Distributions
The handful of shapes real data keeps turning out to have.
- 8Discrete DistributionsBernoulli, binomial, geometric, negative binomial and Poisson, and the variance-to-mean check that tells them apart. Assuming Poisson for Pantry's session counts understates the tail by a factor of 16.
- 9Continuous DistributionsUniform, exponential and normal, standardising and z-scores, and the memorylessness that makes ten days of waiting worth exactly nothing.
- 10The Normal Family: Chi-Square, t and FWhere the chi-square, t and F distributions come from and why each one exists. A nominal 95% interval built with 1.96 covers 81% at n=3, and the t distribution is the exact repair.
- 11Choosing a DistributionHow the families connect, how to pick one, and heavy tails and the lognormal. Pantry's order values fail every normality check; their logarithm passes to four decimals.
Module 4 · From Sample to Population
Why a number computed from 500 people says anything about five million.
- 12Sampling Distributions and Standard ErrorThe sampling distribution of a statistic, and the difference between a standard deviation and a standard error. On Pantry they are 158 times apart, and that factor is exactly the square root of n.
- 13The Central Limit TheoremWhat the CLT promises, how large n has to be before it delivers on a skewed metric, and where it fails entirely. Pantry's order values need n=500, not the n=30 of the rule of thumb.
- 14Estimators and Maximum LikelihoodBias, variance, consistency and mean squared error, then maximum likelihood. Why a constant that ignores your data can beat the sample mean at small n, and why that stops being true as n grows.
- 15The BootstrapResampling your own data to get a standard error for a statistic that has no formula, percentile intervals, and the cases where the method breaks.
Module 5 · Inference
Turning a sample into a claim, and knowing how often the claim is wrong.
- 16Confidence IntervalsWhat 95% confidence actually means, shown by coverage simulation, z versus t, and why the proportion interval everyone learns under-covers.
- 17Hypothesis Testing and P-ValuesNull and alternative, test statistic, rejection region, and what a p-value is and is not. Under the null a p-value is uniform, and that one fact explains the 5% false positive rate.
- 18Type I, Type II and PowerThe two error types, the two overlapping distributions and the areas under them, and how n, effect size and alpha move those areas. At sample sizes teams actually use, 61% of genuine wins come back non-significant.
- 19Comparing Two GroupsTwo means with the pooled and Welch t-tests, two proportions, and paired data. On unbalanced groups the pooled test fires more than half the time when nothing is happening.
- 20Chi-Square Tests for Categorical DataGoodness-of-fit, independence and homogeneity, plus Fisher's exact test and McNemar. Plain chi-square holds its error rate well below the textbook warning, and the usual corrections overshoot.
- 21Nonparametric and Permutation TestsPermutation tests and Mann-Whitney, what each one actually tests, and how a rank test and a t-test can both be right and still disagree.
- 22Bayesian InferencePrior to posterior, the beta-binomial conjugate, and a credible interval versus a confidence interval. A p-value of 0.16 and a 92% posterior probability that treatment wins are not in conflict.
Module 6 · The Statistics Behind an A/B Test
The same machinery, in the place you will be asked about it.
- 23An Experiment as a Two-Sample ProblemThe sampling distribution of a difference, and where an experiment's standard error comes from. Measuring a difference costs four times the traffic of measuring a level.
- 24Choosing a Test for an ExperimentMetric shape to test, the CLT argument that licenses a t-test on revenue, and where that argument runs out. The test name barely matters; the row you analyse doubles the error rate.
- 25Alpha, Power and Sample SizeAlpha, power, MDE and n as one set of four dials that trade against each other, and why the MDE is set by finance rather than by statistics.