Shipping the Decision

By the end of this lesson, you should be able to: turn an experiment readout into a ship, iterate or stop decision using a fixed procedure, explain why the steps go in this order, and write the decision up so others can check it.

The procedure

Each lesson in this track is one step of the same procedure.

Is the experiment valid? (lessons 6 to 8)
What does the primary metric say, compared with what matters? (lessons 4 and 9)
What could this test have detected? (lesson 5)
Did any guardrail get worse? (lesson 4)
Is the result believable? (lessons 9 and 15)
Who does it work for, and will it last? (lessons 16 and 19)
Decide, and write it up.

The order matters. A guardrail result means nothing if the split was broken. A big win needs extra checking before anyone celebrates. The rest of this lesson walks through each step with a real decision.

Step 1: is the experiment valid?

Before reading any metric, check that the experiment worked as designed.

The split. Run the sample ratio mismatch check (lesson 7).
The data. Check data loss, filters and metrics that shouldn't move, by group (lesson 8).
The design. Check that the experiment ran for at least a full week, and that the share of traffic in each group stayed the same throughout.

MSN's carousel experiment shows why this comes first. The readout said 16 cards reduced engagement. The split check said the treatment had lost users. Once the bot filter that removed them was fixed, the result reversed, and the change shipped. Reading the metrics first would have killed a winning feature.

Microsoft's team applied this same standard to every result it reported in its 2014 paper: each experiment ran for at least a week, the allocation was stable, and the sample ratios matched.