Interleaving for Ranking Evaluation

By the end of this lesson, you should be able to: explain interleaving as a paired design, run team-draft interleaving and credit clicks correctly, say what has to be true for it to beat an A/B test, and name what it can't measure.

The problem it solves

You want to compare two ranking algorithms: two ways of ordering search results, recommendations or a feed. An A/B test shows ranker A to one group of users and ranker B to another, then compares.

The difficulty is that users differ from each other far more than rankers do. Some people watch hours of video a day and some barely open the app. A small difference between rankers has to be detected through that enormous spread between people, and the spread has nothing to do with the rankers.

Netflix's 2017 explanation uses a taste test. To find out whether people prefer Coke or Pepsi, you could give one group only Coke and another only Pepsi, then compare how much each group drinks. But soda habits vary hugely, from people who hardly drink any to heavy drinkers. Or you could offer every person both, unlabelled, and see which they choose. That removes "the uncertainty contributed by the wide range in population-level soda-consumption habits".

Interleaving is the second design for rankings. Blend both rankers' results into a single list, show it to one user, and see which ranker's results they choose. Both rankers are judged by the same person at the same moment, so everything about that person cancels out. It's pairing, applied to rankings.

Team-draft interleaving

The usual version is team-draft interleaving, which works like picking teams in a playground:

Toss a coin to decide which ranker picks first.
The rankers take turns. Each adds its highest-ranked result that isn't already in the list.
When the user clicks or plays a result, credit the ranker that contributed it.
Compare the rankers' shares of clicks (Netflix used the share of viewing hours).

The coin toss handles position bias. Users pay far more attention to the first slots than to later ones, so a ranker that always filled slot 1 would win regardless of quality. Netflix's requirement is that "at any given position in a row, a video is equally likely to have come from ranking algorithm A or B."