Salesforce — Cyber Security Detection — A/B Test & Causal Inference Questions

Role context: Data Scientist (SMTS), Cyber Security Operations Center · Est. study time: 80 min · 7 questions

Experimentation in this domain

Security detection breaks most of the statistical machinery you bring from product data science, and knowing which parts break is the skill this section tests.

You mostly cannot run experiments. You cannot randomize being attacked. You cannot randomize a control to half your employees and watch the other half get breached. Real incidents are far too rare to power anything. So the workhorse is not the A/B test, it is shadow mode: deploy a detection without alerting, log what it would have fired on, and adjudicate a sample. It costs nothing, needs no randomization, and protects the only resource that matters, which is the analysts' trust.

You mostly cannot compute recall. The denominator is attacks you missed, and you cannot see them. Every honest evaluation in this domain is a substitute: technique coverage, purple-team catch rate, synthetic injection curves, retrospective hunting.

Six things shape everything below:

• Extreme class imbalance. Malicious events are perhaps 1 in 100,000 or rarer. Accuracy is a number that only flatters you, and a small false-positive rate applied to an enormous negative population swamps the true positives.
• Costs are wildly asymmetric, and the loss function has a cliff. A false positive costs ~20 analyst-minutes; a miss can cost £183m. That arithmetic seems to argue for alerting on everything. It does not, because past a noise level analysts mute the rule, and a muted detector has an effective recall of zero. The constraint is human, not statistical.
• The label is context. In the words of the VP who runs this org: "The same signal represents either normal business or a real attack depending entirely on context." That is why single-signal detection fails and correlation exists.
• Baselines must be per entity. Salesforce profiles behavior per org and per user from roughly three months of history. A global threshold alerts on every power user and misses every small attack.
• Everything is a repeated test. Scoring every user against their own baseline every day is millions of implicit hypothesis tests. "Three sigma" fires constantly at that scale.
• The data adapts. Your model decays because it worked. Low-and-slow exfiltration exists precisely because volume thresholds do. No retraining on historical data anticipates a technique invented last week.

The set below covers the measurement core (operating points, evaluation without labels, baselining, multiple testing), then weights toward signal correlation, the one place a real A/B test is possible, and causal reads where it is not.