Heavy Tails and Capping
By the end of this lesson, you should be able to: say what a heavy tail does and does not break, work out how much of your variance a handful of users owns, cap a metric knowingly rather than by reflex, and argue for a different metric when capping is the wrong answer.
What a heavy tail actually costs
Basket's revenue per user over the free-delivery experiment:
| Mean | $9.83 |
| Median | $0.00 |
| Standard deviation | $26.75 |
| Skewness | 4.29 |
| Users spending nothing | 81.0% |
| Share of all revenue | |
|---|---|
| Top 0.1% of users | 3.0% |
| Top 1% | 17.4% |
| Top 5% | 52.1% |
| Top 10% | 77.8% |
Half the money comes from one user in twenty.
The instinct is that this breaks your test. It doesn't, and lesson 10 measured that: a Welch t-test on this exact column holds its error rate at 5.2%. The central limit theorem does not care that the data is skewed, only that the mean is approximately normal, and at these sample sizes it is.
What the tail costs you is power, and it costs a lot.
One percent of users, forty percent of the variance
Here's the number that makes the problem concrete.
The 679 users above the 99th percentile ($126.10) are 1.0% of the sample. They contribute 39.9% of the total squared deviation from the mean, and squared deviation is exactly what a standard error is built from.
So two fifths of your uncertainty comes from one user in a hundred. Every extra user you recruit is mostly buying you more of the 99% who contribute almost nothing to the variance, which is why revenue tests need so much more traffic than conversion tests on the same population.