Triggered Analysis and Dilution

By the end of this lesson, you should be able to: explain why a feature that works can produce a flat experiment, compute the dilution factor and undo it, run the triggered analysis without introducing a much worse bias, and say what counterfactual logging is for.

A feature that worked and did not ship

Basket built advanced search filters and tested them on 91,000 users. The readout:

	Users	Lift	95% interval	p
Everyone assigned	91,000	+2.41%	[−0.33%, +5.16%]	0.085

Not significant. The team moved on.

Now split those users by whether they ever actually opened the filters. Only 11,126 of them did, which is 12.2%.

	Users	Lift	95% interval	p
Everyone assigned	91,000	+2.41%	[−0.33%, +5.16%]	0.085
Triggered users only	11,126	+24.25%	[+16.11%, +32.40%]	0.0000
Never triggered	79,874	−0.62%	[−3.54%, +2.29%]	0.674

The feature works, and it works enormously. It just doesn't work on the seven users in eight who never went near it, and those users are 87.8% of the average.

Left, the same experiment read on three populations. Right, what a low trigger rate costs in traffic.

Look at the third row for a moment, because it's the check that makes the second row believable. Users who never triggered show −0.62% with an interval straddling zero. That's what you want: a feature nobody saw should do nothing, and here it does. If that row had shown an effect, something would be wrong with the assignment or the logging, and the triggered result couldn't be trusted either.

The dilution is exact arithmetic

This isn't mysterious. If a feature affects only the fraction $r$ of users who trigger it, and does nothing to the rest, then the average effect over everyone is:

$\Delta_{\text{overall}} = r \times \Delta_{\text{triggered}}$

Check it on Basket:


Absolute effect on triggered users	+0.04476
Share who trigger	0.1223
Predicted overall effect	+0.00547
Observed overall effect	+0.00445

Close enough that the identity is clearly what's driving it, with the small gap being sampling noise on a diluted number.

Rearranged, that formula is a warning about power. The effect you're trying to detect shrinks by a factor of $r$ , and from lesson 5 you know sample size scales with $1/\delta^2$ , so the traffic you need scales with $1/r^2$ .

Trigger rate	Effect you would see	Users per arm needed
100%	24.25%	1,181
50%	12.13%	4,721
25%	6.06%	18,882
12.2% (Basket)	2.97%	78,943
5%	1.21%	472,028

Basket's 12.2% trigger rate multiplied the traffic requirement by 67 times. A feature that could have been proven on 1,181 users an arm needed 78,943, and the test ran with 45,616.

That's the whole story of this experiment. It wasn't a failed feature. It was a feature measured on a population that was 88% irrelevant.