Microsoft — Bing — Product Sense & Metrics

Role context: Senior Data Scientist, Bing Growth & Experimentation (Microsoft AI) · Est. study time: 45 min · 5 practice questions

What Is Bing

Bing is Microsoft's web search engine — a query-to-answer engine that takes a user's intent, retrieves and ranks the world's information, and increasingly generates the answer directly through Copilot, the AI chat-and-answer layer woven into the results page. The product's whole job is to satisfy intent as fast and as well as possible. Users pay nothing; Microsoft captures value by keeping them coming back (retention and query share) and by monetizing commercial-intent queries through Microsoft Advertising. So data science optimizes a two-step chain — user success → retention → query volume → commercial queries → ad revenue — under a hard rule that chasing revenue must never degrade the experience that produces the queries.

Two structural facts make search different from the transactional and subscription products most DS folks train on. First, there is no purchase event — whether a search "worked" is inferred from behavior (clicked, dwelled, reformulated, came back, or abandoned), and the best outcomes can look like the worst in the logs. Second, distribution is a confound, not a given: Bing's reach is heavily tied to being the default in Edge, Windows, and Office, not to users freely choosing it the way they choose Google (Bing runs ~5% of global search but far more on desktop). A jump in usage is often a Windows rollout, not a preference shift — so separating default-driven usage from genuine retention is a first-order problem here.

The product is in two stages at once: a mature core search engine (enormous bases, heavy guardrails, marginal gains, rigorous experimentation) bolted to a 0→1 generative-AI surface (Copilot, with metrics that didn't exist a few years ago — answer groundedness, helpfulness, multi-turn chat retention). AI has rewritten the funnel mid-flight: Copilot resolves many queries on the page, lifting satisfaction while cannibalizing ad clicks and the referral traffic publishers depend on. The culture is explicitly data-driven — "changes only ship when their impact is understood and positive" — which is why experimentation, success-metric definition, and forecasting are the center of gravity for this role. (Bing is, in fact, where CUPED and much of Microsoft's experimentation platform were invented.)

User Journey

The core search session is a loop, and the most important moment is invisible — the user who is satisfied without clicking:

The Copilot / AI-answer journey adds new stages on top: answer groundedness and correctness, multi-turn engagement (turns per chat), citation click-through (the publisher's cut), and chat retention.

The decisive, delicate ratios: successful-session rate (intent resolved), reformulation rate (a refinement is fine, a failure is bad — and they look alike), good vs bad abandonment (answered-on-page vs gave-up — identical in click logs), and return rate / sessions per user (the retention signal that actually distinguishes a better engine from a noisier one).

Different Sides of the Market

Search is a three-sided attention market, with the web index as a fourth, supply-side input:

This role primarily optimizes the user side — their success, satisfaction, retention, and engagement. Advertisers are the monetization side, whose value rides on query volume and commercial-query intent. Publishers supply the content and get referral traffic back — which AI answers can quietly starve. The hidden sides candidates miss: the distribution surfaces (Edge/Windows/Office) that drive most acquisition, internal Finance/leadership (this role forecasts business risk and opportunity and feeds the ship-decision process), the human relevance judges who label answer quality, and Trust & Safety / responsible AI (spam, bots, and Copilot hallucination as guardrails on everything).

Metrics by Domain

North Star Metric: successful query share — the share of searches that actually answer the user.

Why this one, and not something simpler? Three plain reasons:

• Not raw query count. When a search fails, people just re-type it — so the count goes up when the product gets worse.
• Not revenue. You can lift revenue today by showing more ads, but that makes search worse and pushes users away tomorrow.
• "Did the search work, and did the user come back?" is the only thing that really shows the engine got better.

So we track successful query share, and we treat revenue as a line we protect, not the goal — grow it, but never by hurting search quality.

North Star in numbers (example values). Let's put rough numbers in so it's concrete:

So if a change lifts successful-session rate from 85% to 86%, that's about 0.75B more successful searches a month. That single point is what you're really trying to move — and it matters far more than making the raw query count look bigger.

Revenue uses the same searches. Ad revenue = queries × RPM ÷ 1000, where RPM is revenue per 1,000 searches. At ~75B searches and an example RPM of $13, that's about $1B a month. You grow it by showing ads on more commercial searches, or by a higher click rate or price per click — but only as long as search quality doesn't drop.

An early-warning number. If about 60% of users come back after four weeks, then retained users × success ≈ 600M × 85% ≈ 510M users who came back and succeeded. This is the number that tells you the engine got better, not just bigger.

The data you'd look at daily

Most days, this role works with huge logs of search activity. Three tables do most of the work (example columns and rows):

1. Search session log — one row per query/impression (billions of rows a day). The raw behavior:

The abandon column (good vs bad) isn't recorded directly — you have to work it out from behavior: if someone reads the answer and leaves happy, that's good; if they give up and re-type or leave, that's bad. Getting this right is a big part of the job.

2. Experiment scorecard — one row per experiment × metric. This is what you read to make a ship call:

3. Daily metric aggregate — one row per day × slice; the source the dashboards roll up from:

What the dashboards look like at Bing

Two views matter most each day for this team.

The health dashboard — checked every morning by the team and leadership:

Your job is to explain every Watch or Alert cell — "why did this metric move?" That "why did it move" work is exactly what this role is hired for (here, the Copilot drop traces back to one model update).

A trend tile sits alongside it — the North Star over recent weeks, where a dip is a story to chase:

The experiment scorecard — this is what you read to decide: ship the change, or hold it? (It's table 2 above.) The rule is simple — ship only if all three are true:

1. The main metric went up (here, successful-session rate is clearly up).
2. No guardrail is broken (here, one is — latency went up 40ms).
3. The traffic split is clean, with no SRM problem.

In this example the search got better, but the latency guardrail failed — so you don't ship yet. First, ask engineering to bring the latency back down, then ship.

Quick check

Practice Questions