Randomize a new encoder by session or by player?

Experiment designHard

Problem. Engineering has a new video encoder that should reduce stalls at the same bitrate. An engineer proposes randomizing at the session level: each session flips a coin, so the same player sometimes gets the new encoder and sometimes the old one. They argue this is more efficient because every player contributes to both arms. How would you design this test?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.