Why Did It Move?

By the end of this lesson, you should be able to: run a "why did this metric move" investigation as a fixed procedure, explain why the order of the steps matters, separate several causes that happen at once, and finish with a recommendation.

The procedure

Every lesson in this track is one step of the same procedure. Here it is in order.

1. Is it real? Compare with a baseline and its normal range (lesson 15).
2. Is the data right? Check the measurement (lesson 17), and take out any known one-off events.
3. Which part moved? Break the metric into its parts: new users and lost users, volume and rate (lessons 4, 11 and 12).
4. Cut by the suspects. List what could have changed and segment by the dimensions that would show it (lesson 14).
5. Size each cause (lesson 16).
6. Recommend what to do.

The case: Netflix, first quarter of 2022

In April 2022, Netflix reported that it had lost 200,000 paid memberships in the first three months of the year.1

Netflix's shareholder letter explains what happened in enough detail to follow every step of the procedure.