Engagement is up, a guardrail slipped, and the split looks off. Ship?

Decide from resultsHard

Problem. A test finished. Engaged members are up 0.5% (p = 0.04). A guardrail, unsubscribes, is up 2% (p = 0.03). The traffic split was set to 50/50 but the data shows 51.2/48.8. The team is reading about 12 metrics. Do you ship?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

← Cut variance on a noisy engagement metric, and analyze a ratio rightA notifications policy shipped platform-wide. Measure it without an A/B. →