Choosing the Test
By the end of this lesson, you should be able to: choose a test from the shape of the metric, explain why the t-test works on data that isn't normal, say how many users a skewed metric needs before the usual tests can be trusted, and spot when two metrics with the same name measure different things.
Start with the shape of the metric
The right test depends less on the name of the metric than on what kind of number it is.
| Kind of metric | Example | Usual test |
|---|---|---|
| A yes/no outcome per user | Converted, retained, clicked at least once | Two-proportion z-test (the same as a chi-square test) |
| An average per user | Sessions per user, revenue per user | Welch's t-test |
| A ratio whose denominator isn't the randomised unit | Clicks per page view, revenue per order | Delta method (lesson 11) |
| A skewed average | Revenue per user with a few big spenders | t-test with enough users, or capping (lesson 12) |
| A percentile | 95th percentile page load time | Methods built for percentiles, or a bootstrap |
| Few randomised units | 20 cities, 50 teams | Permutation test or other small-sample methods |
Most product metrics fall in the first two rows, and for those, the standard tests work well. The other rows each need care, and each has its own lesson or section.
Most of the warnings don't apply
People often worry that a t-test is invalid because "the data isn't normal". Revenue per user is wildly skewed, with most users at zero and a few spending a lot. Surely a t-test can't handle that?
It can, because the t-test doesn't need the data to be normal. It needs the average to be roughly normal. And the central limit theorem says that the average of many independent values is close to normally distributed, whatever the shape of the individual values, as long as there are enough of them.
Microsoft's CUPED paper puts it plainly: online experiments have at least thousands of users per group, so the normality assumption is usually unnecessary.