Population and Sample

By the end of this lesson, you should be able to: tell a population from a sample and a parameter from a statistic, explain why a complete census is still a sample, say what sampling error is and roughly how big it is, and split any question about data into a describing half and an inferring half.

Nothing is broken, and the number is still wrong

Pantry is a weekly grocery app. Last week it took 25,079 orders. Sum the value column, divide by 25,079, and the answer comes back:

$54.29.

That query is right. Every order is in it, none is double counted, the decimal is in the correct place. Put it on a dashboard and nobody will ever find a fault in it.

The average order value at Pantry is $54.53.

We can say that flatly because Pantry's data comes out of a generator we wrote, and the number that generates order values was set by hand before a single row existed. The truth is knowable here in a way it never is at work, which is the only reason this track can grade your answers instead of just teaching you procedures.

So the dashboard is off by 24 cents. Not because of a bug. Because of what a week is.

"But we counted every order"

This is the objection worth taking seriously, because it sounds airtight and it's how most people quietly opt out of statistics.

The reasoning goes: sampling error is what you get when you survey 400 people instead of asking everyone. We didn't survey anyone. We have every order, no gaps. So there's nothing to be uncertain about.

It doesn't hold, and the reason matters.

You have every order that happened. You do not have every order that could have happened. Pantry's checkout doesn't emit a fixed list; it runs a process, and that process threw off 25,079 orders last week the way a die throws off numbers. Run the week again with the same customers, the same prices and the same weather, and you get a different 25,079 orders and a different average. Nobody would have to change anything for that to happen.

A census of last week is a complete count of last week. It is a sample of the process. And the process is what you're actually being asked about, every time somebody says "what's our average order value" and means it as a fact about Pantry rather than a fact about one week.

Keep that separation and most of this track follows from it. Lose it and you'll defend numbers that don't deserve it.

Four words, and interviews turn on keeping them straight

Four terms do all the work. They get used loosely in conversation and precisely in interviews, and the precise versions are worth memorising.

Population. Every unit you want the answer to be about. Here, every order the Pantry checkout process can produce. Usually infinite, usually hypothetical, almost never sitting in a table.

Sample. The units you actually have. Here, last week's 25,079 orders.

Parameter. A number describing the population. The true mean order value, $54.53. Fixed, unknown, and not going to change while you look at it.

Statistic. A number computed from the sample. The $54.29 on the dashboard. Known exactly, and different every time you draw a new sample.

Notation follows the same split, and it's not decoration: Greek letters are parameters, Roman letters are statistics.

$\mu = \text{the population mean} \qquad \bar{x} = \text{the sample mean}$

$\sigma = \text{the population standard deviation} \qquad s = \text{the sample version}$

The whole of inference is the gap between the two columns. You want $\mu$. You have $\bar{x}$. Everything else is machinery for saying how far apart they're likely to be.

Run the week again, 5,000 times

Here's what a generator buys you: we can run last week again. Same process, same 25,079 orders, 5,000 times, and watch where the dashboard lands.

The average of those 5,000 weekly averages is $54.54, which is the truth to within a cent. Individually they scatter: one standard deviation is $0.35, and across 5,000 weeks the extremes ran from $53.31 to $55.86.

Notice what that means for a single week. Only 52.5% of weeks land within 25 cents of the truth. 84.8% land within 50 cents. 99.6% land within a dollar.

Our week's $54.29 is a thoroughly ordinary draw. Not a bad week, not a broken pipeline. One draw.

That scatter has a name. Sampling error is the difference between a statistic and the parameter it estimates, caused by nothing except which units happened to land in your sample. It is not a mistake, and no amount of engineering removes it.

What more data buys, and what it doesn't

Fine, get more data. How much more?

Read down the first two columns together. Ten times the orders cuts the error to roughly a third. A hundred times the orders cuts it to a tenth.

That's the $1/\sqrt{n}$ rule, and on the right-hand panel above the measured line and the $1/\sqrt{n}$ reference line are the same line.1 The practical version is worth saying out loud, because it prices every "can we just run it longer" conversation you'll ever have:

Four times the data, half the error.

Going from 10,000 orders to 40,000 halves your uncertainty. Halving it again costs 160,000. This is why an underpowered experiment can't be rescued by a few more days, and why the last decimal place is always the expensive one. We'll put a formula on it in sampling distributions and spend it properly in Type I, Type II and power.

Describing and inferring are different jobs

One column, two questions, same arithmetic, and only one of them is a guess.

"What was the average order last week?" $54.29. Exact. There's no uncertainty in it at all, because the question is about the 25,079 rows and you have all of them. This is descriptive statistics: summarising what you have.

"What is the average order at Pantry?" $54.29, give or take about $0.67. This is inferential statistics: using what you have to say something about what you don't.

The queries are identical. The second question is the one that needs a range attached, and it's also the question almost everybody actually means.

Most reporting failures at work are this confusion wearing a suit. A weekly number moves from $54.29 to $54.61, somebody writes a paragraph about why, and the honest answer is that two draws from the same process differ by that much three weeks in four. Catching that doesn't need a better dashboard. It needs a number for how big an ordinary wobble is, which is the next four modules.

Takeaways

• A complete census is still a sample. You have every order that happened, not every order the process could produce, and the process is what the question is about.
• Parameters are fixed and unknown; statistics are known and vary. $\mu$ = $54.53 never moves. $\bar{x}$ came in at $54.29 this week and will be something else next week.
• Sampling error is not a mistake. Across 5,000 clean weeks the averages ranged from $53.31 to $55.86, with no bug in any of them.
• A single week is one draw. Only 52.5% of weeks land within 25 cents of the true mean, so a 24-cent miss is the normal case, not the alarming one.
• Four times the data, half the error. Precision costs quadratically, which is why "run it a bit longer" rarely fixes an underpowered result.
• Answer the inferring question with a range. A point estimate with no interval is a description pretending to be an inference.

Quick check

Practice

Take it further

These questions all turn on separating what a number describes from what it's being used to claim.

• Bing: forecasting next quarter's query volume asks for a figure about a period that hasn't happened, which is the purest form of an inferring question.
• Uber: did the TV campaign drive trips? starts from a complete record of trips and still can't answer from description alone.
• Netflix: how many accounts for 0.5pp? is the $1/\sqrt{n}$ trade priced out on a real decision.

Next, summarising a distribution takes the population chart above seriously. The mean and the median were $16.53 apart, and picking the wrong one is a more expensive mistake than any sampling error in this lesson.