Mix Shift and Simpson's Paradox
By the end of this lesson, you should be able to: explain how every segment can improve while the total gets worse, run a shift-share decomposition that closes exactly, tell a within effect from a between effect, and give the right recommendation when the answer is "nothing is broken".
Five channels up, blended rate down
First, one piece of housekeeping. Lesson 10 found a payment regression in android 8.2 that depresses conversion for everybody. We've already located that cause, so it's excluded here. Removing a defect you have already explained before hunting for the next one is ordinary practice, and doing it in the other order means the thing you understand buries the thing you don't.
With that out, here is Basket's order conversion by acquisition channel, first four whole weeks against the last four.
| Channel | Rate before | Rate after | Change | Share before | Share after | Shift |
|---|---|---|---|---|---|---|
| Organic | 22.73% | 23.15% | +0.42 | 37.0% | 30.7% | −6.3 |
| Paid search | 22.63% | 22.83% | +0.21 | 24.8% | 21.3% | −3.5 |
| Paid social | 8.83% | 9.30% | +0.47 | 15.0% | 30.6% | +15.6 |
| Referral | 23.21% | 23.76% | +0.55 | 14.4% | 11.3% | −3.2 |
| Partnership | 23.02% | 23.40% | +0.37 | 8.7% | 6.0% | −2.7 |
| Blended | 20.708% | 18.928% | −1.780 |
Read the change column. All five channels improved. Not one of them got worse.
Now read the bottom row. The blended rate fell by nearly two points.
Both are true. There's no error, no definitional trick, and no missing data. This is Simpson's paradox, and it is the single most common way a competent analyst reaches a confidently wrong conclusion.
Why it happens
Look at the share column. Paid social went from 15.0% of sessions to 30.6%.
Paid social converts at around 9%. Everything else converts at around 23%. The blended rate is a weighted average of the five, so doubling the weight on the worst one drags the average down even while every input rises.
Nothing about the product got worse. Nothing about any channel got worse. The population changed, and the average moved because averages do that.
Any time a rate moves and you can't find a segment that explains it, check whether the segment sizes moved. A blended rate has two moving parts: the values and the weights. Most people only look at one.