Bayesian A/B Testing

By the end of this lesson, you should be able to: produce a Bayesian readout of an experiment, say what it gives you and what it doesn't, show with a number why "check it every day" still breaks, and judge when the prior is worth arguing about.

The question people actually ask

Chris Stucchio ran A/B testing at AOL Patch before moving to VWO, a testing tool. In VWO's 2015 whitepaper on its Bayesian method, he described the question people brought him most often at the end of a test: "what is the probability that variation B is better than variation A?"1

A p-value can't answer that. It's the probability of data at least this extreme if there were no difference, which is a different thing. A Bayesian analysis answers the question directly, by treating each version's conversion rate as uncertain and describing that uncertainty with a probability distribution. Bayesian inference in the Statistics track covers the machinery; this lesson applies it to a test.

A readout, end to end

Take an illustrative test: a one-tap rebooking button, shown to half of a marketplace's users.

The usual test: $z = 2.50$, $p = 0.012$. Significant.

Bayesian, step by step. Give each version's rate a flat Beta(1, 1) prior. Updating is just addition, as the beta-binomial update showed:

$\text{Control: Beta}(1 + 5{,}570,\; 1 + 24{,}282) \qquad \text{Treatment: Beta}(1 + 5{,}867,\; 1 + 24{,}281)$

Comparing the two posteriors gives:

And you can ask about any lift that matters to the decision:

That's what the Bayesian readout adds: a probability rather than a yes/no verdict, and a distribution for the size of the effect that you can read at any threshold. If the change needs a 4% lift to pay for itself, the answer is about a coin flip, even though "significant" sounds decisive.