What a business case is really testing

Hi, I'm Ryan, a senior data scientist at a FAANG company. Over the past few years, I've gone through interview processes at companies like Amazon, Meta, Pinterest, Uber, and TikTok, and was fortunate to receive offers from several of them.

One thing I noticed is that many data science interviews revolve around business case questions, often in industries I'd never worked in before. I didn't get through them by memorizing frameworks. Instead, I learned how to quickly understand a product: how it makes money, who it's built for, and what could go wrong. Then I used that to reason through the problem.

This guide is simply the approach that worked for me. I hope it helps you too.

A business case hands you an open-ended problem and watches how you think. There is rarely one correct answer. The interviewer wants to see whether you can take something vague, break it into parts, and reach a decision without getting lost.

The three kinds of business case

Almost everything you get asked falls into one of three buckets. They sound different, but each one has its own comfortable way of thinking, and the rest of this guide walks through all three in turn.

1. Diagnose a problem. A metric moved and nobody knows why yet.

• Weekly active users dropped 12% over the last month. What happened?
• Checkout conversion on mobile fell two points right after a redesign. Why?
• Support ticket volume has been climbing for six weeks straight. What's driving it?

2. Measure success. Something already shipped, or is being piloted, and you need to say honestly whether it worked.

• We removed the free trial and moved straight to a paid plan. Did that help the business?
• The recommendation algorithm was rebuilt last quarter. How do you know if it's actually better?
• Marketing ran a big campaign last month and signups jumped 20%. Was the campaign worth what it cost?

3. Decide whether to launch. Nothing has shipped yet, and you're being asked whether it should.

• Product wants to add a cheaper subscription tier with ads. Should we build it?
• The team wants to change the default privacy setting for new accounts. How would you decide?
• Someone wants to remove a feature that looks unused. Would you?

Try answering one of these in your head before you read on. You won't have a framework yet, and that's fine. The point is just to notice where the question feels vague and what you'd want to ask first.

Framework 1: Diagnosing a problem

Step 1: Clarify the question

Don't start solving. Spend the first minute making sure you understand the question. For a metric drop, that means pinning down which metric, how large the move was, over what period, and compared to what.

Two things are worth ruling out early on any metric move:

• Is it real? A logging change, a late-reporting data source, or a redefined metric can look exactly like a genuine drop.
• What decision depends on the answer? If nothing changes based on what you find, you are solving the wrong problem.

Clarifying questions aren't stalling. They show you won't sprint off in the wrong direction, which is the most common way these answers fall apart.

Step 2: Break it into parts

Once you know what you are solving, split it into parts before guessing at causes. You want pieces that don't overlap and together cover everything. That is the idea behind MECE (mutually exclusive, collectively exhaustive). In practice this usually means writing the metric as an equation and seeing which piece moved.

A drop in "net adds" is vague. A drop you have traced to failed renewals is almost a diagnosis on its own. Splitting the metric turns one big mystery into a few smaller, checkable ones.

Quick check. A subscription app reports that net new subscribers fell about 15% this month. Before guessing at a cause, how would you break the number down?

Step 3: Segment the data

Breaking the metric into parts is one axis. Cutting the population is the other. A real problem is almost never spread evenly, so the fastest way to find it is to ask where it concentrates. These are the cuts worth running on nearly any metric move:

Two habits make segmentation actually decisive rather than just busywork. First, quantify each segment's contribution: a segment that is down 20% but is only 5% of the total explains one point of a ten-point drop, so keep looking. Second, watch for mix shift. The overall number can move while every segment is flat, simply because the mix of users changed. This trips up experienced people, so it is worth checking explicitly.

Quick check. Signup conversion fell from 6.0% to 5.0% over a quarter. Desktop conversion is still 8%, mobile is still 3%. Neither moved. What happened?

Step 4: Pick the chart that answers the question

In the room you can't pull up a dashboard, but saying which picture you would look at tells the interviewer you have actually done this. Each investigation question has a chart that answers it:

The first row does the most work. A metric that steps down on a single day was almost certainly hit by something you did: a release, a config change, a logging break. A metric that drifts down over weeks is usually the market, the season, or a competitor. Saying "I'd plot the daily series with our release dates marked on it" is one sentence, and it signals more than a paragraph of theory.

Step 5: Form a few hypotheses, then check the cheapest one

With the drop localized, list the plausible causes. A simple split keeps you honest: internal things you did (a release, a price change, a bug, a new checkout flow) versus external things you didn't (seasonality, a competitor, a holiday, a shift in the economy). The fix is completely different depending on which it is, so it is worth separating them out loud.

Then go after the check that would settle it fastest, not the one that is most interesting. If a metric dropped the same day as a release, comparing users on the old and new app versions is a quick query that can confirm or kill the theory. Usually the segments you cut in the last step have already narrowed the suspect list; the job here is to pick the one query or comparison that turns the leading suspect into a confirmed cause.

Quick check. Daily active users on a photo-sharing app dropped 8% starting last Tuesday. Name two internal and two external causes, and the first check you would run.

Step 6: Reach for heavier tools only if the simple cuts don't work

Most cases resolve with decomposition and segmentation. Occasionally an interviewer pushes past that ("every segment looks flat, now what?"), and it helps to have an answer ready.

Changepoint detection. When you're watching hundreds of metrics across many segments, eyeballing every chart stops scaling. A changepoint method finds the exact date a series shifted, which you then line up against the release and campaign log. It automates the "when did it start" question, nothing more.

A quick classifier as a lead generator. If the drop won't localize to any obvious cut, label affected users versus unaffected, train a simple model to tell them apart, and read which features it leaned on most. If "payment method" or "app version" tops the list, you have a suspect you hadn't thought to cut by. Be honest about what this is: it ranks suspects, it doesn't prove causes. You still confirm with a direct comparison.

Step 7: Land on a recommendation

An analysis with no decision attached is trivia. Close every case by connecting what you found to what you would do. A good ending names the finding, the one check that would confirm it, and the action if it holds:

The drop is concentrated in the latest Android release and lines up with a checkout change. I'd confirm with a version comparison, and if it holds, roll that change back while we fix it.

If you aren't certain, say what you would need to become certain. Being clear about your confidence beats pretending to a precision you don't have.

Framework 2: Measuring whether it worked

Here something already shipped, or is being piloted, and you have to judge it honestly. This is where a lot of candidates go wrong, because they report the number that looks good instead of the number that answers the question.

Step 1: Start from intent

What was the change for? Success is the metric that reflects that goal, counted honestly and net of its cost. If a change was meant to raise revenue, don't grade it on engagement, and vice versa.

Step 2: Think in incrementals, not gross numbers

The word that matters is incremental. Gross numbers flatter you. If you waive fees to win new sellers and 10,000 sign up, the question is not how many signed up. It is how many signed up because of the waiver who wouldn't have otherwise, minus the fees you gave up from the ones who would have joined anyway. And always name a guardrail for the side that could quietly get worse.

Quick check. A marketplace waived its commission on sellers' first month to attract new sellers. Signups rose 30%. Did it work?

Step 3: Pick the right way to measure the effect

The cleanest answer comes from a randomized test: split people into two groups, only one sees the change, and any difference between them is the effect. That isn't always possible, so it helps to know the fallback options too, roughly in order of how much you can trust them:

Move down the table only when you have to. Each step down trades some rigor for feasibility, and saying that out loud, instead of pretending a before-and-after is as good as a test, is exactly what a strong answer sounds like.

If the question is which channel or touchpoint deserves the credit, the same ladder applies. Last-touch attribution is simple but over-credits whatever sits at the bottom of the funnel. Splitting credit across the whole path is more honest. And when the stakes justify it, holding out a channel entirely and measuring what disappears is the most convincing answer of all.

Quick check. A food-delivery app raised its service fee in every city on the same day. Revenue per order rose, but order volume dipped. Leadership asks whether the fee change helped or hurt overall. You can't run an experiment after the fact. What do you propose?

Step 4: Check that it lasts

A win in the first week can fade. Anything new gets a bit of a bump just for being new, called a novelty effect, so a good read watches the metric for a few weeks, not just the first few days, to see if the lift holds once the novelty wears off.

Some outcomes you actually care about take a long time to show up. Retention six months out, or how much a customer is worth over their lifetime, can't be read from a two-week test. In that case, you measure something that shows up early and is known to predict the long-term outcome, a surrogate metric, and confirm with the real outcome once enough time has passed.

Step 5: Put a number on it and call it

Close it the same way you'd close a diagnosis: name the incremental effect, net of its cost, against the goal you started from, and say what you'd do next, whether that's scale it, hold it, or roll it back.

Framework 3: Deciding whether to launch

Now there's no result yet. You're being asked whether to build and ship something. The strongest answers don't just weigh pros and cons, they describe the experiment that would actually settle it, end to end.

Step 1: Get the hypothesis straight

Write one sentence: "Changing X will improve Y, because Z." If you can't write it, you're not ready to test it.

"Adding a cheaper, ad-supported tier will grow paid subscribers, because it captures price-sensitive people who never subscribed at the full price."

Step 2: Pick your metrics

You need two kinds. The primary metric is the one number that decides success, chosen up front so you don't go fishing after the fact. The guardrail metrics are the things that must not get worse: a change can win on the headline number and quietly hurt the business elsewhere.

Step 3: Choose the randomization unit

Decide what you actually split, usually the user rather than the visit, so the same person always sees the same version. Pick a unit big enough to contain spillover: if one person's treatment can affect someone else, like a shared account or a marketplace where the two sides meet, split by the bigger unit even though it costs some statistical power.

Step 4: Decide how many people you need and for how long

Smaller effects need far more people to detect reliably, so the size of the effect you're hoping for drives the size of the test. Run for at least one or two full weeks so you cover weekdays and weekends, not just a lucky few days. If you need to detect a small effect without a huge sample, a technique called CUPED can shrink the sample you need without changing what the test measures.

Step 5: Run it without peeking

Launch and let it reach the size you planned. The hard rule is not to stop early just because the numbers look exciting; early results bounce around a lot, and stopping the moment they look good is the single most common way to call a fake winner. If you genuinely need to watch a test continuously, use a method built for that, sequential testing, instead of just peeking at a normal test.

Step 6: Read the results and decide

Compare the groups, check the guardrails, and make the call: ship, don't ship, or iterate.

Quick check. Product wants to add a "save for later" button to the shopping cart. How would you decide whether to launch it?

When you can't fully randomize

Some launches can't be split cleanly by user: a price change across a whole market, a brand campaign, a policy that has to apply everywhere at once. When that happens, shrink the unit instead of giving up on testing altogether. Launch in one or two markets first, and compare them against markets that didn't get the change, the same quasi-experiment methods from the measuring-success framework. Expand only once that comparison looks good.

Quick check. A streaming service with one $16/month plan is considering a cheaper $8/month plan with ads. How would you test this before rolling it out everywhere?

What separates a strong answer

• You quantify each branch before going deep on one, so effort lands where the numbers are, not where the story is fun.
• You separate what you caused from what happened to you when diagnosing. The fix is different for each.
• You count the incremental effect, not the gross one, when judging whether something worked.
• You name the trade-off and the guardrail out loud instead of optimizing one number in a vacuum.
• You describe the actual experiment, metrics and all, when asked to decide on something new, instead of just debating pros and cons.
• Every number ends in a decision. A figure with no action attached is decoration.
• You're honest about what you can't cleanly measure, and reach for a comparison group instead of pretending everything can be A/B tested.

Common mistakes

• Jumping to a cause before breaking the metric down and segmenting.
• Stopping at the first plausible story without checking how much of the move it explains.
• Reporting gross impact when the honest number is incremental.
• Forgetting the guardrail, so a "win" hides a harm somewhere else.
• Designing a launch decision around opinions instead of describing the test that would settle it.
• Ending on an analysis instead of a decision.

A quick recap

Each type leans on its own sequence. Run the one that matches the question:

Diagnose:  Clarify → Decompose → Segment → Chart → Hypotheses (internal vs external) → Cheapest check → Recommend
Measure:   Intent → Incremental, not gross → Pick the right method → Check it lasts → Call it
Launch:    Hypothesis → Metrics → Randomization unit → Size & duration → Run without peeking → Decide

Further reading

• Decode and Conquer, Lewis C. Lin. Product and case frameworks.
• Ace the Data Science Interview, Nick Singh & Kevin Huo. Business cases and experimentation.
• Trustworthy Online Controlled Experiments, Kohavi, Tang & Xu. The measurement and incrementality side.
• The Pyramid Principle, Barbara Minto. Where the MECE and issue-tree idea comes from.