Interleaving for Ranking Evaluation

By the end of this lesson, you should be able to: explain interleaving as a paired design, run team-draft and credit clicks correctly, say what has to be true for it to beat an A/B test, and name what it cannot measure.

The problem it exists for

Meridian wants to compare two ranking models for its search surface. Ranker B is genuinely +3.5% better.

The obstacle is that sessions differ from each other far more than the rankers differ from each other:

An A/B test shows ranker A to one group of sessions and B to another, then compares. It has to detect a difference of 0.007 through noise of 0.12. It is fighting the between-session variation, and that variation has nothing to do with the rankers.

Interleaving removes it. Blend both rankers' results into one list, show that list to one session, and see which ranker's contributions get clicked. Now both rankers are judged inside the same session, so everything about that session, the query, the user, the time of day, applies equally to both and cancels.

That's pairing, and it's the same mechanism that took a paired t-test from 14% power to 99.8% in the statistics track.

Team-draft, and the credit rule

The version to know is team-draft interleaving, and it's a schoolyard team pick.

Alternate which ranker fills the next slot. Randomise who picks first in each session. When a user clicks a result, credit whichever ranker contributed it. At the end of the session, whichever ranker got more clicks wins that session.

Randomising who picks first is what handles position bias. Top slots get far more attention than lower ones, so if one ranker always filled slot 1 it would win regardless of quality. Alternating and randomising the order means each ranker gets the top slot half the time.

Run it on 20,000 Meridian sessions:

Among the 11,898 decisive sessions, preference for B is 0.5192 ± 0.0090, which is $z = 3.98$ against the null of 0.5.

Notice that 41% of sessions tie and carry no information, exactly as McNemar found: only the discordant pairs say anything.