Acquisition: Where Users Come From
By the end of this lesson, you should be able to: compare acquisition channels without the comparison being decided by how long each has been running, separate a volume result from a value result, say where in the funnel a bad channel actually goes wrong, and know which of these questions needs an experiment rather than a query.
A campaign that worked, by the only number anyone checked
This is the New term from the growth equation, opened up. Somebody at Basket turned on a large paid social campaign in the middle of April, and here is what happened to signups.
Weekly signups went from 1,656 to 3,222. Ninety-five percent growth in six weeks. If your acquisition dashboard shows total signups, and most do, this is the best quarter the team has ever had.
Look at the composition though.
| Channel | Before | After | Change | Share before | Share after |
|---|---|---|---|---|---|
| Paid social | 560 | 2,329 | +316% | 33.8% | 72.3% |
| Organic | 506 | 420 | −17% | 30.5% | 13.0% |
| Paid search | 320 | 315 | −1% | 19.3% | 9.8% |
| Referral | 182 | 126 | −31% | 11.0% | 3.9% |
| Partnership | 89 | 32 | −64% | 5.4% | 1.0% |
One channel tripled. Every other channel shrank in absolute terms, and paid social went from a third of new users to nearly three quarters.
Nothing here tells you whether that was a good trade. To know that, you need to ask what a user from each channel is actually worth, and that question has a trap in it.
The tenure trap, which decides most channel comparisons before you start
Here's the query somebody will run: total orders per user, grouped by acquisition channel, over all time.
That query is wrong, and it's wrong in a direction you can predict.
A user who signed up in March has had eleven weeks to place orders. A user who arrived in May has had two. Group by channel and you aren't ranking channels, you're ranking how long each channel has been running. Any channel that was big in March looks excellent. Any channel that scaled in May looks terrible. Change nothing about the product and the ranking flips as the mix ages.
The fix is to hold tenure constant. Give every user the same clock, starting from their own signup:
- Pick a window. Fourteen days works for a product with weekly rhythm like Basket.
- Measure each user only over their own first fourteen days.
- Drop anyone who hasn't had a full fourteen days yet, rather than counting them short.
That last rule matters more than it looks. Including a user who is six days old, with six days of orders, drags down whichever channel is growing fastest. It's the same bias in a smaller costume.
Say the words "I'd hold tenure constant and compare the first N days from signup" in an interview and you've separated yourself from most candidates in one sentence. It's the single most common unforced error in acquisition analysis.