Novelty and Primacy Effects

By the end of this lesson, you should be able to: explain novelty and primacy effects, tell a real change in the effect over time from a statistical illusion, describe how users' long-term learning can reverse a short-term result, and decide how long a test needs to run.

Two short-term effects

Microsoft's experimentation team describes two effects that can make the first weeks of an experiment misleading, in opposite directions.1

Novelty effect. When something new appears, some users explore it out of curiosity, clicking around more than they normally would. The treatment looks better than it really is, and the effect fades once the novelty wears off. If the feature isn't truly useful, it can fade to nothing.

Primacy effect. When you change something users are used to, like the navigation of a website, experienced users are slower at first because their habits no longer fit. The treatment looks worse than it will be, and it improves as users adapt.

A real novelty effect

MSN changed the Hotmail link on its home page so that it opened Hotmail in a new tab instead of the same one.2 Users noticed. On the first day the change went out to everyone, 20% of all feedback messages were about it, mostly negative. By week two, it was 4%, and by weeks three and four, 2%.

The engagement data had the same shape: a strong early effect, as users were surprised and tried the link several times. But the effect didn't disappear. Among users who clicked the Hotmail link, clicks per user across the whole home page rose 8.9%, a result that held up when Microsoft repeated the experiment on 2.7 million users. (This is clicks on the home page overall. The count of clicks on the Hotmail link itself was inflated by the logging problem in lesson 8.) The feature stayed.

So a novelty effect doesn't mean the change is worthless. It means the early number overstates it, and you need to see where it settles.