Your Feed scorecard has 60 metrics — how many 'wins' are fake? — Meta Facebook Data Scientist Interview

Your Feed scorecard has 60 metrics — how many 'wins' are fake?

Multiple testingHard

Problem. Every Feed experiment reports a scorecard of ~60 metrics. This experiment shows 3 metrics "significant" at p < 0.05, the rest flat. The team is excited about the 3 wins. Should they be? How do you read a big scorecard honestly?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"Were these 3 metrics pre-registered as the primary hypotheses, or are they the 3 that happened to pop out of 60?"
"Are the 60 metrics independent, or highly correlated (many are)?"
"Are the 'wins' in the primary metric family, or scattered secondary metrics?"

Lay out your approach

Do the false-positive math. At α = 0.05, testing 60 truly-null metrics yields about 3 false 'significant' results by chance alone. So "3 of 60 significant" is exactly what you'd expect if the feature did nothing. That reframes the excitement.
Separate primary from exploratory. A pre-registered primary metric significant at 0.05 is meaningful; 3 significant metrics cherry-picked from 60 secondaries are likely noise. The fix starts with declaring the primary metric before looking.
Apply a multiple-testing correction to the scorecard — Bonferroni for a hard family, or Benjamini-Hochberg to control the false discovery rate more powerfully.
Look for coherence. Real effects usually move a cluster of related metrics in a consistent story, not 3 unrelated ones.

Then check in: "Three significant out of sixty is roughly the false-positive rate at 0.05, so I'd distinguish the pre-registered primary from exploratory hits and correct for multiple testing. Want me to do the expected-false-positive math?"

Simulate the back-and-forth

Interviewer

Walk me through why 3 of 60 isn't exciting.

You

If all 60 metrics were truly unaffected, each still has a 5% chance of a false positive. Across 60 tests, the expected number of false positives is 60 × 0.05 = 3. So seeing exactly 3 "significant" is the null expectation — completely consistent with the feature doing nothing. The probability of at least one false positive across 60 independent nulls is 1 − 0.95^60 ≈ 95%, so some "wins" are almost guaranteed. The 3 hits aren't evidence until we account for how many shots we took.

Interviewer

So how do you correct for it?

You

Two standard options. Bonferroni tests each metric at 0.05/60 ≈ 0.00083 — very strict, controls the chance of any false positive, but low power. Benjamini-Hochberg controls the false discovery rate — the expected fraction of your 'discoveries' that are false — which is more powerful and usually the right choice for a big scorecard where you can tolerate some false positives among many. Under either, a p = 0.03 hit from the scorecard almost certainly won't survive. I'd also just report the primary metric's result separately at its own α, since that's the hypothesis we actually pre-committed to.

Interviewer

The metrics are mostly correlated. Does that change things?

You

Yes. Bonferroni assumes independence and is conservative when metrics are correlated, so it over-corrects. With 60 correlated metrics the effective number of independent tests is smaller, so the true false-positive count is a bit below 3, and a method like Benjamini-Hochberg (which handles positive correlation reasonably) or a permutation-based correction is more appropriate. But the core point stands: judged against the number of tests, 3 scattered hits are not a green light — I'd want the primary metric to move, ideally with a coherent cluster of related secondaries.

Land a clear conclusion

You

My read: 3 of 60 significant is the false-positive rate, not a win. I'd insist on the pre-registered primary metric as the decision metric, apply a false-discovery-rate correction to the scorecard, and look for a coherent story across related metrics rather than isolated hits. If the primary didn't move and only 3 scattered secondaries did, I'd call it a null result and not ship on cherry-picked noise. The scorecard is for monitoring and hypothesis generation, not for harvesting significance.

Key summary — what really matters.

60 tests at α = 0.05 → ~3 false positives expected under a true null.
Pre-register the primary metric; scorecard hits are exploratory.
Correct for multiplicity — Bonferroni (strict) or Benjamini-Hochberg (FDR, more powerful).
Correlation makes Bonferroni conservative; FDR/permutation fits better.
Look for coherent metric clusters, not isolated significant metrics.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Framing & estimand	Whether you separate primary from exploratory.	You treat all 60 metrics equally.	You anchor on a pre-registered primary metric.
Design validity	Whether you know the false-positive math.	"3 wins, ship it."	You show ~3 false positives are expected at α = 0.05 over 60.
Statistical rigor	Whether you apply a correct multiplicity fix.	No correction.	Bonferroni vs Benjamini-Hochberg, with correlation caveat.
Decision & trade-offs	Whether you demand coherence.	Ship on scattered hits.	Require the primary + a coherent metric cluster.
Communication	Whether you reframe the excitement honestly.	"Three metrics are significant."	"That's the noise rate; here's the real decision metric."

Interview tips

Strong-answer signal: computing the expected false positives (m × α), insisting on a pre-registered primary, applying an FDR correction, and demanding a coherent metric story. Common trap: celebrating cherry-picked significant metrics from a large scorecard with no multiplicity correction.