The update gets more thumbs-up — but is it just sycophancy?

Metric designHard

Problem. A candidate model update raises thumbs-up rate and session length. You suspect it's more agreeable/flattering rather than more helpful — the sycophancy risk. How do you tell, and what's your call?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Design an A/B test for a new model version — what's the OEC?It wins on offline evals but loses online — which do you trust? →