Reading a Result: P-Values and Confidence Intervals
By the end of this lesson, you should be able to: say what a confidence interval and a p-value mean and don't mean, separate a statistically significant result from one that matters, calculate the chance that a significant win is false, and read an experiment's result step by step.
What a result reports
An experiment readout usually gives three numbers for each metric.
- The estimated effect. For example, conversion up 1.2%.
- A confidence interval. For example, from +0.4% to +2.0%.
- A p-value. For example, 0.004.
Each one answers a different question.
The confidence interval
A 95% confidence interval is a range of effects that are consistent with the data. If you ran the same experiment many times, and built an interval the same way each time, about 95% of those intervals would contain the true effect.
The interval is usually the most useful number in the readout, because it shows two things at once.
- Where the effect probably is. An interval from +0.4% to +2.0% says the change probably helps, by somewhere between a little and a fair amount.
- How precise the experiment was. A narrow interval means a precise estimate. A wide one means you don't know much yet.
Compare it with two lines: zero, and the smallest effect that would matter to the business. An interval entirely above the business threshold is a clear win. An interval entirely below zero is a clear loss. An interval that straddles both means you can't tell yet.