Bayesian A/B Testing
By the end of this lesson, you should be able to: produce a Bayesian readout of a real experiment, say what it buys and what it doesn't, kill the optional-stopping claim with a number, and know when a prior is worth arguing about.
Meridian runs a test
Meridian is a ride-hailing marketplace. It ships one-tap rebooking to half its riders and measures whether they complete a trip.
| Arm | Conversions | Users | Rate |
|---|---|---|---|
| Control | 5,570 | 29,852 | 18.66% |
| Treatment | 5,867 | 30,148 | 19.46% |
Frequentist: , , 95% CI on the absolute difference . Significant. Ship it.
Bayesian, with flat Beta(1,1) priors on both arms:
| Quantity | Value |
|---|---|
| P(treatment > control) | 0.9937 |
| Median lift | +4.30% |
| 95% credible interval | [+0.91%, +7.80%] |
Same data, and the second readout tells you three things the first doesn't. It gives a probability rather than a verdict. It gives the effect's distribution rather than a point and an interval you're not allowed to interpret. And it lets you ask about any threshold you like:
| Threshold | P(lift exceeds it) |
|---|---|
| 0% | 0.9937 |
| 1% | 0.9718 |
| 2% | 0.9072 |
| 4% | 0.5676 |
| 6% | 0.1677 |
The planted truth is +4.0%, and the posterior puts 56.8% of its mass above 4%. That's what a well-calibrated belief looks like.
The claim that brings teams here
Almost every team that considers switching says some version of this: a posterior is valid at any moment, so we can watch it live and stop when it's convincing. No more waiting two weeks.
The first half is true. A posterior is a correct summary of the evidence so far, at every moment, with no correction needed. That is a genuine difference from a p-value, whose interpretation is tied to a fixed sample size.
The second half does not follow, and this is the measurement that settles it.
Run Meridian's experiment with both arms identical. There is no effect. Check P(B > A) once a day and stop the first time it crosses a threshold. Every stop is a false positive:
| Threshold | 1 look (fixed horizon) | 14 daily looks | 28 daily looks |
|---|---|---|---|
| P(B > A) > 95% | 4.9% | 19.7% | 24.2% |
| P(B > A) > 99% | 1.0% | 5.4% | 6.2% |
At a fixed horizon the 95% rule fires 4.9% of the time, which is exactly right.
Checked daily for two weeks, it fires 19.7% of the time.
That is the same inflation a p-value suffers, in the same direction, at roughly the same magnitude. The Experimentation track measures 5.6% becoming 19.0% for the frequentist version of this experiment. The two frameworks are equally vulnerable because the vulnerability was never in the framework.