Databricks — Product & Growth — A/B Test & Causal Inference Questions

Role context: Data Scientist (Senior/Staff), Data Team · Est. study time: 55 min · 5 questions

How experimentation works here

Enterprise-SaaS experimentation is subtler than consumer A/B:

  • The unit is often the account (few, heterogeneous, whale-skewed consumption), so account-level A/Bs are underpowered — variance reduction and quasi-experiments matter.
  • Many levers (a sales/CS motion, a pricing change) can't be randomized across accounts → observational causal.
  • Adoption/expansion causality is confounded by selection — engaged accounts adopt more.
  • Churn is consumption contraction, and you must prove an intervention causally reduces it.

For the fundamentals — p-values, power, error types, distributions — see the Probability & Statistics section.

Each answer is a coaching walkthrough: a Sample answer (clarify → approach → a simulated back-and-forth → a clear call), then a Deep dive with illustrative example, then a Grading rubric.

Questions (5)

A/B a platform feature with few, whale-skewed accountsExperiment designHardMeasure a sales/CS motion you can't randomize across accountsCausal inferenceHardYour consumption metric is dominated by a few whalesVariance & ratio metricsMediumDoes the churn-prevention play actually reduce churn?Causal inferenceHardWhich accounts does the feature actually help?Heterogeneous effectsHard