Product Analytics
Define a metric properly, then use it to work out what moved and why, on one product's real data.
20 lessons in 5 modules, all running on one product's data and building on each other. Start at the top.
0 of 20 done
Module 1 · Foundations
What a metric is, which one to pick, and what its distribution is hiding.
- 1Define the Metric Before You Move ItFour teams counted daily active users on the same table and got four different answers, the biggest 52% above the smallest. Here's where the gap comes from and how to write a definition nobody can argue with.
- 2Which Metric Should This Business Care AboutBasket's last quarter was either a 72% triumph or a 26% decline, depending on which number you put on the slide. Two tests separate a real North Star from one you can simply buy.
- 3The Growth EquationOne identity ties acquisition, retention, churn and resurrection together. Split Basket's growth with it and the headline story changes completely: the business isn't growing faster, it's leaking faster.
- 4Distributions: Why User Data LiesBasket's average user places 1.94 orders. Three in five users place fewer than that, and the average describes almost nobody. Here's why user data always comes out this shape, and what it breaks.
Module 2 · The Growth Engine
The five states a user moves through, and the rate that governs each one.
- 5Acquisition: Where Users Come FromBasket doubled its weekly signups. It also tripled the size of its worst channel and starved its best one. Here's how to compare channels honestly, starting with the tenure trap that makes every naive comparison wrong.
- 6Activation: The First SessionBasket's most impressive activation metric predicts 87.0% retention and reaches 2.1% of new users. Here's how to find a real activation moment, and why the best-looking one is almost always a trap.
- 7Retention CurvesThe same Basket cohort retains at 11%, 42% or 70% on day 30, depending which standard definition you use. Then the cohort triangle shows something the aggregate curve hides completely.
- 8Churn: Defining It Before Predicting ItSeven defensible inactivity thresholds give Basket seven churn rates, from 5% to 59%. The data can price each choice for you, but it can't make it, and pretending otherwise is the most common mistake here.
- 9Resurrection and Win-BackBasket brings back nearly twice as many users as it acquires, and each one is worth half as much. Resurrection is the largest term nobody budgets for, and the only one measured without a denominator.
Module 3 · Decomposition
Splitting a number until the piece that moved is the only one left.
- 10Funnel Analysis and Drop-offBasket's worst funnel step loses 44% of users and is perfectly healthy. The step worth fixing lost 14 points and hides in a segment you only find by splitting two dimensions at once.
- 11Additive and Multiplicative DecompositionBasket's GMV grew $123,483. Two of the three standard ways to split that between users, frequency and basket size don't even add up, and the piece they leave behind is 17% of the change.
- 12Mix Shift and Simpson's ParadoxEvery one of Basket's five acquisition channels improved its conversion rate. The blended rate fell 1.78 points. Nothing is broken, nobody made a mistake, and the dashboard is red.
- 13Segmentation: Choosing the CutThe same 1.78-point decline, decomposed four ways. Three of them say the segments got worse. One says nothing got worse at all. Every one closes exactly, and the data won't tell you which is right.
Module 4 · Investigation
Working out what a number should have been before deciding that it fell.
- 14Forecasting and BaselinesBasket's Saturday runs 37% above its Tuesday. Until you know what a number should have been, you can't say it fell, and a one-day drop under 5% here means nothing at all.
- 15Sizing the OpportunityBasket has two problems worth $79k and $76k a week. They are the same size and they are not remotely the same opportunity, and knowing why is the difference between an analyst and someone who gets listened to.
- 16Data Quality: Doubt the Instrument FirstBasket lost 72% of its iOS search events for three weeks and nobody noticed, because the number still went up. The diagnostic that catches it takes one query and works even when nothing looks wrong.
- 17Why Did It Move?Basket's checkout conversion fell 4.5 points. The answer isn't one thing, it is three, they pull in different directions, and running the steps in the wrong order gets you the wrong answer twice.
Module 5 · Decision
Turning the analysis into money, credit and a recommendation you can defend.
- 18Attribution ModelsFive standard attribution models, scored against a known truth. They disagree with each other by under one point and they are all wrong by fifty. The argument the industry has been having isn't the one that matters.
- 19Monetization: LTV, Payback and the Formula That LiesThe textbook LTV formula says a Basket user is worth $121 for life. They hit $119 in eight weeks and were still spending $21 a fortnight. The formula isn't slightly off, it is structurally wrong.
- 20Growth Loops and ViralityBasket's viral coefficient is 0.066, which sounds like failure and is worth 7% free growth. It also varies twelve-fold by acquisition channel, which is why the April campaign halved it.