Microsoft — Ads Identity — A/B Test & Causal Inference Questions

Role context: Senior Applied Scientist, Ads Data Management Platform · Est. study time: 70 min · 6 questions

How experimentation works here

Experimentation on identity infrastructure is unusual:

  • You often can't randomize identity (a person either resolves or doesn't), so the graph's ads impact is measured with incrementality (ghost ads / PSA, holdout audiences), not a raw A/B.
  • Ground truth for identity is missing, so estimating precision/recall is itself a statistics problem (deterministic holdouts, panels, capture–recapture) with truth-set bias.
  • A probabilistic graph needs calibration, not just ranking — a match score must mean what it says to be thresholded per use case.
  • The unit of randomization is subtle: a person's devices can land in different arms, contaminating an A/B of two matching models.
  • Signal loss from cookies/consent is missing-not-at-random, so measurement leans on modeling the gap.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (6)

You can't randomize identity — how do you prove the graph lifts ads?Experiment designHardEstimate the graph's precision and recall with no ground truthMeasurement & estimationHardYour match model has AUC 0.95 but ships badly — why calibration?Measurement & estimationHardA/B two matching models — but a person's devices split across armsInterferenceHardA consent change hit coverage — measure its true impactCausal inferenceHardHow many samples to trust a 3% false-merge rate?Power & sample sizeMedium