Triggered Analysis and Dilution

By the end of this lesson, you should be able to: explain how analysing everyone dilutes the effect of a change that reaches only some users, analyse only the triggered users correctly, explain why control needs counterfactual logging, and report a triggered result.

When a change reaches only some users

Many changes only affect users in a particular situation.

A new checkout design only matters to users who reach checkout.
A new answer box only appears for certain search queries.
A fix for a rare error only affects users who hit the error.

Users who never reach that situation get exactly the same experience in treatment and control. Their behaviour can't be changed by the experiment. Including them in the analysis adds their noise and none of the effect.

The dilution is simple arithmetic

Suppose 10% of users reach the new checkout, and for them the change increases purchases by 5%. The other 90% are unaffected. Across everyone, the effect is roughly:

\text{Effect across everyone} \approx \text{Share triggered} \times \text{Effect on triggered users} = 10\% \times 5\% = 0.5\%

A 0.5% effect needs vastly more users to detect than a 5% one. Since the users needed grow with one over the effect squared (lesson 5), diluting the effect tenfold multiplies the users needed by a hundred. Analysing only triggered users needs fewer users entering the experiment overall, even though you still have to wait for the 10% to trigger.

Try it: analyse everyone, or only the users the change can reach

Share of users who trigger the change10%Effect on users who trigger it+5%

Illustrative: a metric whose standard deviation equals its mean, 80% power at 5%. Users entering is everyone assigned, whether or not they trigger.

Effect seen across everyone

+0.50%

diluted from +5%

Users needed, everyone

640,000

per group

Users needed, triggered only

64,000

per group, 10.0× fewer

With 10% triggering and a 5% effect, analysing only triggered users needs ten times fewer users entering the experiment. In general, the saving is one over the trigger share.