Sampling and the Central Limit Theorem

Why we sample: you can't measure everyone

Suppose you want to know one number: the average daily screen time of every university student in Taiwan — roughly 1.2 million people. You will never measure all of them. Reaching every student would cost a fortune, take months, and by the time you finished, the answer would have changed. This is not a special problem; it is the normal situation. Almost every interesting question is about a group too big, too expensive, or too inconvenient to examine in full.

So we do the only sensible thing: we look at a small slice and reason about the whole. The whole group we actually care about is the population (母體) — here, all 1.2 million students. The slice we manage to measure is the sample (樣本) — say, 1,000 students we successfully survey. Measuring everyone is called a census; almost always we settle for a sample instead.

It helps to name two kinds of numbers. A number that describes the whole population is a parameter — for example the true population mean, written with the Greek letter mu (μ). A number we compute from our sample is a statistic — for example the sample mean, written x-bar (x̄). The parameter is fixed but unknown; the statistic is something we can actually calculate. The whole game of this guide is: how well does the statistic x̄ stand in for the parameter μ we cannot see?

The sample mean: add up the n measurements you collected and divide by how many there are. If 1,000 students average 4.7 hours, then x̄ = 4.7. It is our best single guess at the unknown μ.

Using x̄ to say something about μ — using the sample to make a claim about the population — is exactly what statistical inference (統計推論) means. Everything in this track, from confidence intervals to A/B tests, is built on this one move. And the whole move only works if two questions have good answers: is the sample a fair picture of the population, and how far off might our single number be? This guide answers both.

A good sample: representativeness beats size

In 1936 a popular American magazine, the Literary Digest, mailed out a poll to predict the presidential election. They collected an astonishing 2.4 million replies and confidently forecast a landslide for the challenger, Alf Landon. Franklin Roosevelt then won 46 of 48 states. Around the same time, George Gallup polled only about 50,000 people and called the result correctly. How did a tiny poll beat a giant one?

Because the giant sample was crooked. The magazine drew names from telephone directories and car-registration lists — in 1936, the marks of the wealthy, who leaned toward Landon. The sample was huge but systematically tilted away from the population it claimed to describe. This is sampling bias: when the way you pick your sample makes it differ, on purpose or by accident, from the population. When the selection method itself is the culprit it is also called selection bias (選擇偏誤).

The cure is randomness. In a simple random sample, every member of the population has the same known chance of being picked, decided by a coin flip or a random-number generator rather than by who is easy to reach. Randomness has a beautiful property: it does not play favorites. On average it pulls in the rich and the poor, the loud and the quiet, in the proportions they actually occur. That is what makes the sample a fair miniature of the whole.

Most real-world disasters come from skipping this. A convenience sample grabs whoever is handy — your classmates, people walking past one café. A voluntary response sample lets people opt in, like an online poll, which over-collects those with strong opinions. Both feel like data, but they quietly answer a different question than the one you asked.

The sampling distribution: the result is itself random

Now the idea that unlocks everything. You surveyed 1,000 students and got x̄ = 4.7 hours. But suppose a colleague, on the same day, randomly surveyed a different 1,000 students. They would not get 4.7 exactly — they might get 4.6, or 4.9. The sample mean is not a fixed fact about the world; it depends on which people happened to land in your sample. In other words, x̄ is itself random.

So picture repeating the survey not twice but thousands of times — each time a fresh random 1,000 students, each time recording the x̄ you get: 4.70, 4.62, 4.88, 4.75, 4.69, and so on. Now make a histogram of all those x̄ values. That histogram is the sampling distribution (抽樣分配) of the mean: the pattern of how your estimate bounces around from sample to sample. You only ever get to draw one sample in real life — but imagining this distribution of all the samples you could have drawn is what lets us say how trustworthy your one number is.

Two facts about the sampling distribution will carry us a long way. First, if the sampling is unbiased, the sampling distribution is centered right on the true μ — your estimates miss high and low, but they cluster honestly around the truth. That is what makes x̄ a good estimator (估計量), and why the single value you report is called a point estimate (點估計). Second, the spread of that cluster — how tightly the x̄ values huddle around μ — tells you how much your one estimate is likely to be off. Pinning down that spread is the job of the next section.

The standard error and why it shrinks like 1/√n

The spread of the sampling distribution has its own name: the standard error (標準誤). In plain words, the standard error is the typical distance between your sample estimate and the truth. A small standard error means your x̄ is probably close to μ; a large one means your single number could be well off. It is the headline measure of how much to trust an estimate.

Do not confuse it with the standard deviation. The standard deviation (標準差) measures how spread out individual people are — some students use their phones for 1 hour, others for 9. The standard error measures how spread out the averages are across samples — and averages are far steadier than individuals, because a sample's lucky highs and unlucky lows cancel out. Same Greek root, completely different jobs: standard deviation is about people, standard error is about your estimate.

Introduced in words first: take the population's standard deviation σ (how spread out individuals are) and divide it by the square root of the sample size n. Bigger samples give a smaller standard error — but only as fast as √n grows.

Put numbers on it. Suppose screen time has a standard deviation of σ = 2 hours. With n = 100 students the standard error is 2/√100 = 0.20 hours. Bump up to n = 400 and it becomes 2/√400 = 0.10. Go all the way to n = 1,000 and it is about 0.063. Notice the pattern: to cut the standard error in half — from 0.20 to 0.10 — you did not double the sample, you had to quadruple it, from 100 to 400. That is the square-root law, and it is why precision gets expensive fast.

One honest catch: the formula uses σ, the population standard deviation, which we almost never know — if we knew that much about the population we might already know μ. In practice we plug in the sample's own standard deviation, written s, to get an estimated standard error. It is an estimate of an estimate's wobble, but with a decent sample it works well.

The version you actually compute: replace the unknown σ with s, the standard deviation of the data you collected. The little hat means estimated.

import numpy as np

screen_time = df["hours"]                 # one column, the 1,000 students you surveyed
x_bar = screen_time.mean()               # the point estimate, e.g. 4.7
s     = screen_time.std(ddof=1)          # the sample standard deviation, e.g. 2.0
se    = s / np.sqrt(len(screen_time))    # the standard error, e.g. ~0.063

print(f"estimate {x_bar:.2f} hours, standard error {se:.3f}")

The Central Limit Theorem, demonstrated

We now know the sampling distribution is centered at μ and has spread σ/√n. But what shape is it? Here comes the most useful theorem in statistics, the Central Limit Theorem (中央極限定理, often shortened to CLT). Its claim is almost too good to believe: no matter how weird and lopsided the population is, the sampling distribution of the mean becomes approximately a bell curve — a normal distribution (常態分配) — as the sample size grows.

Watch it happen with dice. A single fair die is flat: 1 through 6 are equally likely — nothing bell-shaped about it. Roll two dice and average them, and the middle values (around 3.5) become more common than the extremes, because there are many ways to average to 3.5 but only one way to average a 1. Average thirty dice per roll, repeat many times, and the histogram of those averages is a clean bell curve — even though the thing you started with was perfectly flat. The averaging itself manufactures the bell.

In symbols: for a large enough n, the sample mean behaves like a normal distribution centered at the true mean μ, with a spread (the standard error) of σ/√n. The three big ideas of this guide — center, spread, shape — all live in this one line.

import numpy as np
rng = np.random.default_rng(0)

pop   = rng.exponential(scale=4.0, size=1_000_000)   # a VERY skewed population
means = [rng.choice(pop, size=100).mean() for _ in range(10_000)]

# pop is lopsided, but the histogram of `means` is a tidy bell — that is the CLT.
print(np.mean(means), np.std(means))   # ~the true mean, ~ sigma / sqrt(100)

How large is large enough? A common rule of thumb is n ≥ 30 for mildly skewed data, but treat it as folklore, not law. Nearly symmetric populations look normal almost immediately; severely skewed or heavy-tailed ones may need hundreds. The CLT also has a close cousin, the law of large numbers (大數法則): that one promises x̄ homes in on μ as n grows (the estimate becomes accurate), while the CLT describes the precise bell-shaped way x̄ scatters around μ along the journey (the shape of the wobble). Together they explain why the normal distribution shows up so relentlessly, and why so many statistical procedures can lean on it.

What the CLT does and does NOT promise

The Central Limit Theorem is powerful precisely because it asks so little of the population. But that power gets oversold, so here is the honest fine print — the misreadings that cause real mistakes.

1. It does not make your data normal. Only the distribution of the mean (or sum) turns into a bell. Individual incomes, wait times, and screen-time values stay as skewed as they ever were — do not expect a histogram of your raw 1,000 values to look normal.
2. It needs finite variance. If a population is so heavy-tailed that its variance is effectively infinite (some wealth and insurance-loss models behave this way), the bell never forms and the σ/√n rule breaks. The CLT is a promise about ordinary-tailed populations.
3. It needs independent observations. If your data points clump or depend on each other — students from the same dorm, repeated visits by one user — your effective sample size is smaller than n, the true standard error is larger than σ/√n, and your confidence will be falsely high.
4. It is about means and sums, not everything. The maximum, the minimum, or a ratio of two estimates follow their own, different rules. Do not assume every statistic you compute lands on a bell.

From sampling to the rest of inference

Three ideas from this guide — the sampling distribution, the standard error, and the normal shape from the CLT — are the engine under almost every statistical method you will meet. Once you know your estimate behaves like a bell centered at μ with width σ/√n, you can answer the practical question everyone actually wants answered: how far might the truth be from my number?

The normal curve has a handy feature: about 95% of its mass sits within two standard errors of the center. Flip that around and you get the recipe for an interval that captures the truth most of the time.

The estimate, give or take about two standard errors — the seed of a 95% confidence interval. With x̄ = 4.7 and SE ≈ 0.063, this is roughly 4.7 ± 0.13 hours. That ±0.13 is the margin of error.

That single line grows into the next two guides. Wrap the estimate in its standard error and you get a confidence interval (信賴區間), reported with a margin of error (誤差範圍). Ask instead whether your number is surprisingly far from some claimed value, and you are doing hypothesis testing (假設檢定). Both are just the sampling distribution, read in two different directions.

And when a formula like σ/√n does not apply — a messy statistic, a small or strange sample — there is a beautifully direct alternative, the bootstrap (自助法). Instead of trusting a formula for the sampling distribution, you simulate it: repeatedly resample your own data with replacement, recompute the statistic each time, and let the computer draw the sampling distribution for you. It is the same idea as this whole guide — many samples reveal the spread — done by brute force.

1. We sample because measuring an entire population is impossible — we use a statistic (x̄) to stand in for an unknown parameter (μ).
2. A sample only represents the population if it is chosen randomly; representativeness beats size, and bias never shrinks with more data.
3. The estimate is itself random; the pattern of its variation across samples is the sampling distribution.
4. Its spread is the standard error, σ/√n, so precision improves only as fast as the square root of the sample size.
5. Thanks to the Central Limit Theorem, that sampling distribution is approximately normal — which is the foundation for confidence intervals and hypothesis tests, coming next.