Choosing the Test
By the end of this lesson, you should be able to: pick a test from the shape of your metric, say which choices actually change your answer and which are cosmetic, spot the moment your analysis unit stops matching your randomisation unit, and notice when two people quoting the same metric name are quoting different numbers.
Start with the shape, not the name
Every metric you'll test is one of a few shapes, and the shape picks the test.
| Shape | Example at Basket | Reach for |
|---|---|---|
| Proportion | Did this user order? | Two-proportion z-test, or chi-square |
| Mean of a continuous value | Revenue per user | Welch t-test |
| Count | Orders per user | Welch t-test on the counts |
| Ratio of two sums | Clicks per impression | Delta method, or bootstrap |
That's the table. Now here's the part nobody tells you, which is that most of these choices matter far less than their reputation suggests.
Most of the warnings are wrong
The usual story is that picking the wrong test wrecks your error rate. It's worth checking rather than repeating, so here's the check: take Basket's real data, split it at random into two groups that differ in nothing at all, run each test, and count how often it declares a winner. Three thousand times, six thousand users per arm.
Nothing is happening in any of these splits. Every rejection is a false positive, and a correctly calibrated test should produce them 5% of the time.
| Metric | Test | False positive rate |
|---|---|---|
| Conversion rate | Two-proportion z-test | 4.7% |
| Conversion rate | Welch t-test on the 0/1 column | 4.7% |
| Revenue per user | Welch t-test | 5.2% |
| Clicks per impression | Delta method, user is the unit | 5.1% |
| Clicks per impression | Mean of each user's own CTR | 5.1% |
| Clicks per impression | z-test, impression is the unit | 6.9% |
Look at rows one and two. Running a t-test on a column of zeroes and ones is the thing people get told off for, and it lands on 4.7%, identical to the "proper" proportion test. At this sample size they're the same test wearing different clothes.
Row three is more surprising. Basket's revenue is badly skewed, with a skewness of 4.3 and the top 1% of users holding 17.4% of all revenue. That's exactly the distribution you're warned never to t-test. At six thousand users per arm it comes out at 5.2%, because the central limit theorem doesn't care what the underlying distribution looks like once is large enough. It's the mean that has to be normal, not the data.
So the honest summary is that under the null, most reasonable tests behave. If you were expecting this lesson to be a list of tests that blow up, it isn't, and believing that list is itself a mistake: it makes people spend their attention on the wrong thing.
This does not mean the shape is irrelevant. Heavy tails matter enormously for power, which is how many users you need to detect a real effect. Lesson 12 is about exactly that. What they don't do here is break the error rate.