Expected Loss and the Decision Rule

By the end of this lesson, you should be able to: compute expected loss from a posterior, set a region of practical equivalence from real costs, explain why P(better) is not a decision rule, and defend a shipping threshold with money rather than convention.

The number that isn't a decision

Last lesson ended with P(treatment > control) = 0.9937 on Meridian's rebooking test. As close to certain as an experiment gets.

So ship it?

Here are three results, all of them over 94% likely to be positive:

The third case is 23 times more consequential than the first, and its probability is lower.

P(B > A) collapses the entire posterior into one bit: which side of zero is most of the mass on. It throws away how far, which is the only thing that determines what shipping is worth.

Expected loss

The fix is to keep the magnitude. Expected loss is the average amount you give up by taking an action, integrated over the whole posterior.

$\mathbb{E}[\text{loss from shipping}] = \mathbb{E}\big[\max(0,\; -\delta)\big]$

Read it as: in the worlds where treatment is worse, how much worse, weighted by how likely each of those worlds is. When treatment is better, the loss from shipping is zero, so those draws contribute nothing.

On Meridian:

Across 14 million riders a year at $2.35 of margin a trip, the downside of shipping this is two hundred dollars. The downside of not shipping it is a quarter of a million.

That asymmetry is the decision. Not the probability, the two losses side by side.