Shipping the Decision
By the end of this lesson, you should be able to: run a full readout in the right order, tell an inconclusive result from a null one, weigh a proven harm against an unproven benefit, and write the recommendation in a form somebody can act on.
The brief
Basket built a rewards programme. Points for orders, a visible balance, a redemption screen. It ran for three weeks on 79,330 users.
Nobody is going to tell you what's in this experiment. Here's the workflow, in the order that saves you the most wasted effort: validity first, then the primary, then the guardrails, then the interpretation. Checking validity last means potentially analysing a broken experiment in depth.
Step 1: is the experiment valid?
Before looking at any outcome, check that the comparison is a comparison.
| Check | Result | |
|---|---|---|
| Sample ratio | 39,640 / 39,690, chi-square 0.03, p = 0.859 | PASS |
| Pre-period revenue | $109.17 vs $108.68, p = 0.675 | PASS |
| Pre-period orders | 2.104 vs 2.101, p = 0.863 | PASS |
| Pre-period sessions | 6.139 vs 6.150, p = 0.700 | PASS |
| Tenure | 70.5 vs 69.6 days, p = 0.253 | PASS |
Clean. The randomiser did its job, nothing was filtered after assignment, and the two groups were the same before the experiment started. Whatever we find now is caused by the treatment.
That paragraph is worth writing out in a real readout. It's the licence for everything that follows, and lesson 7's EXP-03 is what happens when nobody checks.