Long-Term Holdouts and Effect Decay

By the end of this lesson, you should be able to: tell a fading effect from a lasting one, say when a holdout can first rule out pure novelty, size a holdout against what it costs, and explain why a long-running holdout stops measuring what you meant.

The short-term effect isn't the long-term effect

Most experiments run for a week or two. What a company actually cares about is what happens over months, after users have got used to the change.

Henning Hohnhold, Deirdre O'Brien and Diane Tang at Google put the problem plainly in 2015: "the short-term effect is not always predictive of the long-term effect, i.e., the final impact once the product has fully launched and users have changed their behavior in response."1

Two kinds of change happen after launch:

• Novelty fades. Users try something because it's new, and the interest wears off. A big early lift shrinks, sometimes to nothing.
• Users learn. Google studied "ads blindness": when people see low-quality ads, they gradually learn to ignore ads altogether, so showing more ads can lose clicks over months even if it gains them in the first week. The reverse, "sightedness", happens when ads are consistently useful.

Both are invisible in a two-week test.

What Google found, and what it changed

To measure user learning, Google ran experiments for months and used methods such as the post-period method: after a long treatment period, give both groups the same experience again and measure how differently they still behave. That leftover difference is what users learned.

The finding was that ads blindness and sightedness are driven by the quality of the ads people had seen before. Google used it to make two changes: an adjustment to its ad auction that gave ad quality more weight, and a 50% reduction in the number of ads shown on mobile search. Cutting the mobile ad load "was a substantially short-term revenue negative change", but measured over the long term, its revenue impact "was shown to be neutral", with a clear gain for users. A two-week test would have killed it.