Old model vs new model on the same backtest: which test?

Test selectionMedium

Problem. You've scored the same set of past applications with the current Trust Index model and a candidate. Each model flags its riskiest 5%. Among 1,300 labeled frauds, the current model catches 494 and the candidate 572. A colleague runs a two-proportion z-test on 38% vs 44%. On the crypto subset the candidate looks worse, but the same test says it's not significant. Is that the right test?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Fraud that gets through is 0.5% of users. Size a test to detect a 20% drop.A stricter step-up cuts fraud and loses good users. Ship it? →