Define success for launching ChatGPT Memory

Measure successHard

Problem. ChatGPT is launching Memory (the assistant remembers facts about you across chats). Leadership asks how you'd define success. What metrics do you propose?

Before you reveal: say your answer out loud, as if you were in the real interview — get your reasoning across clearly first. There is no single correct answer: reading what the interviewer is really after and defending your own thinking is what makes an answer strong.

Clarify the question first

"What's Memory's job — better, more personalized answers, and higher retention? Or a standalone feature to adopt?"
"Who's it for — power users, or everyone? And what's the privacy/trust risk?"
"Is success adoption, or downstream value (better answers, retention)?"

Lay out your approach

Anchor on the outcome Memory is supposed to improve, not adoption. Memory exists to make answers more relevant and the assistant stickier, so success is better answers and higher retention for users with Memory, not "how many turned it on."
Metric chain: adoption (Memory on, memories saved) → behavior (more personalized, resolved interactions; fewer re-explanations) → outcome (retention lift, satisfaction).
Measure incrementally with an A/B: Memory on vs held-back, comparing helpfulness proxies and retention.
Guardrails, heavily: privacy/trust (creepiness, wrong or unwanted memories, opt-outs, complaints), and safety of stored info. A retention win that spooks users is a loss.

Then check in: "Success is Memory improving answers and retention, not adoption counts, measured with an A/B on helpfulness and retention, with strong privacy/trust guardrails. Want me to lay out the metric chain?"

Simulate the back-and-forth

Interviewer

Isn't high adoption of Memory the success metric?

You

Adoption is necessary but not success — it's a leading indicator. Memory could be widely enabled and still not help, or even annoy people with irrelevant recall. What matters is whether users with Memory get more relevant answers (fewer re-explanations, higher task completion) and, downstream, retain better than a held-back control. So I'd treat adoption and memories-saved as leading signals and require them to connect to a helpfulness and retention outcome.

Interviewer

How do you measure "better, more personalized answers"?

You

Behaviorally and qualitatively. Behaviorally: do Memory users re-explain context less (fewer "as I told you, I'm vegetarian" corrections), resolve tasks in fewer turns, and regenerate less on personalized queries? Comparatively: A/B Memory-on vs held-back on those helpfulness proxies and on retention. Qualitatively: survey whether the personalization feels helpful vs creepy, and human-rate a sample of personalized answers. The privacy dimension means I can't judge this on behavior alone — a user might keep using it while feeling uneasy, so I need the sentiment read.

Interviewer

What would make you hold the launch?

You

If the privacy/trust guardrails move the wrong way — spikes in opt-outs, complaints about wrong or unwanted memories, or survey signals of creepiness — even with a retention lift, I'd hold or scope it down (better controls, transparency, easy deletion). The asymmetry matters: the retention upside is bounded, but a trust violation on personal data is expensive and slow to reverse. Ship only if helpfulness and retention improve and trust guardrails hold.

Land a clear conclusion

You

So success isn't adoption; it's Memory making answers more relevant and lifting retention for users who have it, measured with an A/B on helpfulness proxies and retention, validated qualitatively for the personalization-vs-creepy dimension, and gated on strong privacy/trust guardrails. Ship if helpfulness and retention rise with trust intact; hold or scope down if the guardrails move, given the asymmetric cost of a trust violation.

Key summary — what really matters.

Success = better answers + retention for Memory users, not adoption counts.
Metric chain: adoption → personalized/resolved interactions → retention.
Measure incrementally (Memory-on vs held-back A/B).
Privacy/trust guardrails have weight — opt-outs, creepiness, wrong memories.
Validate qualitatively (survey/UXR) — personalization can feel creepy even if used.

Incremental read. A/B Memory-on vs held-back, per user:

Metric	Held-back	Memory on	Read
Re-explanation corrections / session	0.40	0.22	−45% (more personalized)
Task completion	71%	75%	+4pp
D30 retention	62%	64%	+2pp
Opt-outs / complaints	baseline	+1.5pp	guardrail ⚠

Memory reduces re-explaining and lifts completion and retention — a real win — but the opt-out/complaint tick is the trust guardrail to watch. If that guardrail stayed flat, this is a clean ship; if it spiked, the retention lift wouldn't justify it.

Why adoption alone fails. Adoption could be 60% while retention is flat (Memory recalled irrelevant things). Retention is the outcome; adoption is a step toward it. The A/B against a held-back control is what isolates Memory's causal effect from the general trend.

Dimension	What interviewers look for	Below the bar ❌	Above the bar ✅
Problem framing	Whether success is outcome, not adoption.	"Adoption = success."	Better answers + retention; adoption is leading.
Methodology and math	Whether you build a metric chain + A/B.	One metric.	Adoption → personalized/resolved → retention, via held-back A/B.
Business judgment	Whether you weight privacy/trust.	Ignore the creepiness risk.	Trust guardrails with veto; asymmetric-cost reasoning.
Deployment thinking	Whether you validate qualitatively.	Behavior only.	Survey/UXR for personalization-vs-creepy.
Communication	Whether you give a gated recommendation.	"Ship if adopted."	"Ship if helpfulness + retention up and trust holds."

Interview tips

Strong-answer signal: tying success to helpfulness + retention (not adoption), an incremental A/B, and strong privacy/trust guardrails validated qualitatively. Common trap: declaring success from adoption/memories-saved, or ignoring the trust dimension of a memory feature.