Heavy Tails and Capping

By the end of this lesson, you should be able to: explain why a few extreme users make a metric hard to test, cap a metric and say what that changes, choose a cap without biasing the result, and pick other metrics that avoid the problem.

What a heavy tail costs

In many products, most users spend nothing, and a few spend a lot. Revenue per user, purchases per user and minutes watched often look like this. The distribution has a heavy tail: a long stretch of rare but very large values.

Those rare users dominate the metric's variance. One user who spends $5,000 in a week moves the average of a group of 10,000 users by 50 cents, and whether that user lands in treatment or control is luck. That luck shows up as noise, and noise means you need more users.

It hits twice.

The sample size goes up. You need about $16\sigma^2/\delta^2$ users per group (lesson 5), and a heavy tail makes the standard deviation σ large.

The usual tests take longer to become reliable. Lesson 10 showed Microsoft's rule of thumb: you need about 355 × skewness² users per group before a confidence interval can be trusted. Bing's revenue per user had a skewness of 18.2, so it needed 114,000 users per group just for the test to be valid. Microsoft saw skewness above 30 for revenue per customer at a commerce site.

Capping

Capping (also called winsorizing) sets a maximum value for the metric. Any user above the cap is counted at the cap.

Bing capped revenue per user at $10 per user per week. Two things happened.

Skewness dropped from 18 to 5.3, so the number of users needed for a valid test fell from 114,000 to 9,700 per group.
The test became more sensitive. With the same number of users, capped revenue could detect a change 30% smaller than uncapped revenue.

Try it on a simulated revenue metric where 90% of users spend nothing.

Try it: cap a revenue metric

Cap each user's revenue at this percentileNo cap

Illustrative: 90% of users spend nothing, the rest spend a log-normal amount. Users needed use 16σ² ÷ δ² for a 5% lift in the mean.

Skewness

24.9

needs 220,196 users per group for a normal mean

Users per group for a 5% lift

400,685

Revenue changed by the cap

0.0%

of total revenue

With no cap, this metric needs about 400,000 users per group to detect a 5% lift. Capping at the 99.9th percentile cuts that to about 236,000, and changes about 8% of total revenue. Capping harder saves more users but changes more of the metric: at the 99th percentile, about 31% of total revenue is above the cap.