Expected Loss and the Decision Rule

By the end of this lesson, you should be able to: compute expected loss from a posterior, set a region of practical equivalence from real costs, explain why P(better) isn't a decision rule, and defend a shipping threshold with money rather than convention.

A probability isn't a decision

Bayesian A/B testing ended with a rebooking test that was 99.4% likely to be better than control. As close to certain as an experiment gets. So ship it?

Here are three results, all more than 94% likely to be positive:

The third is 23 times more consequential than the first, and its probability is lower.

P(B better) squeezes the whole posterior into one bit: which side of zero most of it sits on. It throws away how far, which is what decides what shipping is worth.

Expected loss

The fix is to keep the size. Expected loss is the average amount you'd give up by taking an action, averaged over everything the posterior says could be true:

$\mathbb{E}[\text{loss from shipping}] = \mathbb{E}\big[\max(0,\; -\delta)\big]$

Here $\delta$ is the true difference, treatment minus control. Read it as: in the worlds where treatment is actually worse, how much worse, weighted by how likely each of those worlds is. When treatment is better, shipping loses nothing, so those worlds count as zero.

VWO built its Bayesian testing tool around exactly this.1 Its test runs until the expected loss of choosing a version falls below a "threshold of caring", which the whitepaper describes as a number "so low that one does not care if an error is made by this amount."

For the rebooking test, with an illustrative 14 million users a year and $2.35 of margin per conversion:

The downside of shipping is about two hundred dollars. The downside of not shipping is about a quarter of a million. That comparison is the decision, not the probability.