Population and Sample

By the end of this lesson, you should be able to: tell a population from a sample and a parameter from a statistic, say what sampling error is and how it shrinks as the sample grows, explain why a bigger sample can't fix a biased one, and say when a complete dataset is still a sample.

Four terms

Every month, the US Bureau of Labor Statistics reports how many jobs the economy added. It doesn't count every job in the country. It surveys about 119,000 businesses and government agencies, covering about 622,000 worksites, and works out the national figure from them.1

That one report contains all four terms this track is built on.

Population. Every unit you want the answer to be about. Here, every job in the US outside farming.

Sample. The units you actually measured. Here, the jobs at the 622,000 surveyed worksites.

Parameter. A number that describes the population. Here, the true change in the number of jobs that month. It's a fixed number, and nobody knows it.

Statistic. A number you compute from the sample. Here, the figure in the headline. You know it exactly, and a different sample would have given a different one.

The notation keeps the two sides apart. Greek letters are parameters. Latin letters, or a letter with a hat, are statistics.

Inference is the step from one column to the other: using the statistic you have to say something about the parameter you don't.

Sampling error

Had the bureau surveyed a different 119,000 businesses, the headline would have come out a little different. Nothing would have gone wrong. The sample would just have been different.

The gap between a statistic and the parameter it estimates, caused only by which units happened to land in the sample, is called sampling error. It isn't a mistake, and no amount of careful engineering removes it. It's the price of measuring part of a population instead of all of it.

What you can do is measure how big it is. The bureau publishes its own: the monthly change in jobs is accurate to about ±122,000, with 90% confidence.1 Its own worked example takes a reported gain of 50,000 jobs and gives the range it's consistent with: anything from a loss of 72,000 to a gain of 172,000.

So a month reported as "+50,000 jobs" might have been a month when the economy lost jobs. The headline number is a statistic, and it comes with a range.

How big sampling error is

How far a statistic typically lands from the parameter is called its standard error. For a mean, it depends on two things:

• $\sigma$, how spread out the individual values are in the population;
• $n$, the number of units in the sample.

For a yes-or-no question, where a share $p$ of the population would say yes, the spread of individual answers is $\sigma = \sqrt{p(1-p)}$.

The widget below runs one survey 500 times. Each time it asks a fresh random sample of people, from a population in which 40% would say yes. The chart shows where the 500 estimates land.

With 400 people per survey, the estimates centre on 40%, but they spread out. The typical survey misses by about 2.4 points, and only 65% of surveys land within 2 points of the truth. Every one of them was done correctly.

Now move the sample size up one step at a time. Each step is four times as many people:

Each row has four times the people of the row above, and half the error. Going from 400 to 1,600 people cuts the miss from 2.4 points to 1.2. Halving it again costs 6,400 people.

Four times the data, half the error. That's what the $\sqrt{n}$ in the formula means in practice. It's why the last decimal place is always the expensive one, and why an underpowered experiment can rarely be rescued by running it a few more days.

Notice what isn't in the formula: the size of the population. A random sample of 1,000 people measures a country of 300 million about as precisely as a city of 1 million. People often assume a sample has to be some decent fraction of the population to be trusted. It doesn't. What matters is how many units you sampled and how they were chosen.

A bigger sample can't fix a biased one

Everything above assumes the sample was drawn at random. When the way units get into the sample is related to what you're measuring, the sample has sampling bias, and a bigger sample doesn't help.

In 1936, the Literary Digest magazine mailed about 10 million postcard ballots for the US presidential election, and nearly 2.4 million came back.2 It predicted Alf Landon would win 54% of the vote to Franklin Roosevelt's 41%. Roosevelt won with 61%.

The usual explanation is that the mailing lists came mostly from telephone books and car registrations, which in 1936 meant wealthier, more Republican voters. Dominic Lusinchi checked that story against a 1937 Gallup survey, and the evidence pointed elsewhere. Telephone and car owners favoured Roosevelt too. The problem was who sent the ballots back. Among people who said they'd received one, nearly 75% of Landon supporters returned it, against 57% of Roosevelt supporters. Only 24% of the ballots came back, and the people who answered weren't like the people who didn't.2

The same thing happened in 2021, at a far larger scale. Two big US surveys tracked how many adults had received a first COVID-19 vaccine dose: Delphi-Facebook, with about 250,000 responses a week, and the Census Bureau's Household Pulse, with about 75,000.3 By May 2021, compared with the official count from the Centers for Disease Control and Prevention, they overestimated uptake by 17 and 14 percentage points. Because the samples were so large, their margins of error were tiny, so they were very confident and very wrong. A survey run by Axios and Ipsos, with about 1,000 responses per wave and careful sampling, tracked the official figure well.

The researchers showed that the 250,000-person survey was no more accurate than a simple random sample of 10 people.3

Go back to the widget and set the response slider to 1.3, so a "yes" person is 1.3 times as likely to answer as a "no" person. The surveys now centre on about 46%, not 40%. Push the sample size to the top: the spread collapses, every survey lands near 46%, none lands within 2 points of the truth, and each one reports a margin of error of ±0.31 points.

More data makes a biased survey more confident, not more right. The margin of error only measures sampling error. It says nothing about bias.

Product data has the same problem whenever people choose whether to be counted. In-app surveys, app-store reviews, users who opt into a beta, customers who answer a satisfaction email: each is a sample of the people who chose to respond, and those people differ from the ones who didn't.

A complete dataset can still be a sample

"We don't survey anyone. We have every order, every session, every user. There's nothing to sample."

It depends on the question.

Say you have every order your company took last week. "What was the average order value last week?" has an exact answer. The population is last week's orders, you have all of them, and there's no sampling error. That's descriptive statistics: summarising the data you have.

"What's our average order value?" is a different question. It's about the business, and the business will keep taking orders. Last week's orders are one week's output from a process that will produce next week's orders too, and next week's won't be in your table. Run the same week again, with the same customers and prices, and people would still order slightly differently. So last week is a sample of the process. That's inferential statistics: using the data you have to say something about data you don't.

The statistician W. Edwards Deming drew this line in 1953. He called the first kind of question enumerative: it's about the units in front of you. He called the second analytic: it's about the process that produced them, and what it will produce next.4

Almost every number that feeds a decision is the second kind. A metric is quoted so that someone can plan, compare or predict, and every one of those is about the future. An A/B test is the clearest case. It includes every user who visited during the test, and you still calculate a p-value, because the decision is about the users who'll come after it.

So when an average moves from one week to the next, the first question isn't "what caused it?" It's "is the move bigger than the normal week-to-week wobble?" Without the standard error, there's no way to tell a real change from a different draw.

Takeaways

• A parameter describes the population and is unknown. A statistic is computed from the sample and changes with the sample. Inference uses the second to say something about the first.
• Sampling error isn't a mistake. The US jobs report is accurate to about ±122,000 jobs, so a reported gain of 50,000 is consistent with a loss.
• The standard error is $\sigma/\sqrt{n}$. Four times the data halves the error, and the size of the population doesn't enter into it.
• A bigger sample can't fix a biased one. The Literary Digest had 2.4 million ballots and called the election for the loser. A 250,000-person weekly survey did no better than 10 people picked at random.
• A complete dataset is still a sample when the question is about the process that produced it, which is almost always.

Quick check

Practice

Take it further

These questions all turn on separating what a number describes from what it's being used to claim.

• Bing: forecasting next quarter's query volume asks for a figure about a period that hasn't happened yet.
• Uber: did the TV campaign drive trips? starts from a complete record of trips and still can't be answered by description alone.
• Netflix: how many accounts for 0.5pp? prices out the $1/\sqrt{n}$ trade on a real decision.