LLN and CLT

Why this matters

Two theorems explain why statistics works at all. The law of large numbers (LLN)The result that the sample mean of independent draws gets closer to the true mean as the number of draws grows. says averages settle down to the true mean as you collect more data. The central limit theorem (CLT)The result that for large n the sample mean is approximately normal with SD σ/√n, whatever the shape of the original distribution (if its variance is finite). says the error in that average is approximately normal, with a size you can calculate. Every confidence interval, t-test, A/B test and insurance reserve you will build leans on them. In this lesson you will check both by simulation rather than taking them on trust.

The law of large numbers

Let X1, X2, ..., Xn be independent draws from the same distribution with mean μ. The sample mean

X̄n = (X1 + X2 + ... + Xn) / n

gets closer and closer to μ as n grows. Flip a fair coin 10 times and you might see 70% heads; flip it 10,000 times and you will be very close to 50%.

The LLN is about the average, not about individual outcomes. It does not say that tails becomes "due" after a run of heads (the gambler's fallacyThe false belief that a run of one outcome makes the other outcome "due"; early streaks are diluted, not corrected.). Early streaks are not corrected; they are diluted by the many flips that follow.

How fast? The standard error

If each draw has standard deviation σ, the sample mean has standard deviation

SE(X̄n) = σ / sqrt(n)

This standard errorThe standard deviation of the sample mean, σ/√n; it describes uncertainty in the mean, not spread of individual values. shrinks with the square root of n. To halve the error you need four times as much data; to cut it by 10 you need 100 times as much. That single fact explains why survey sample sizes, lab replicates and simulation run counts look the way they do.

The central limit theorem

The CLT adds the shape. For large enough n, whatever the original distribution (as long as it has a finite variance):

X̄n is approximately Normal(μ, σ / sqrt(n))

The original data can be skewed, lumpy or discrete. The averages still come out bell-shaped. That is why the normal distribution appears in places where the raw data is not normal at all.

Worked example: averaging skewed waiting times

Waiting times between calls are exponential with mean 1 minute (and SD 1 minute). A single waiting time is very skewed: most are short, a few are long. Now average 30 of them, and repeat that 5,000 times:

import numpy as np

rng = np.random.default_rng(42)          # fixed seed: same numbers every run
draws = rng.exponential(scale=1.0, size=(5000, 30))
means = draws.mean(axis=1)               # one mean per row

print(means.mean())                      # about 1.00 (LLN)
print(means.std())                       # about 0.18 = 1 / sqrt(30) (standard error)

A histogram of means looks close to a normal curve centred on 1 with SD about 0.18, even though a histogram of single draws looks like a steep slide. Try n = 2 and the histogram of means is still clearly skewed; by n = 30 it is nearly symmetric. A quick numeric measure is skewnessA number measuring asymmetry of a distribution; 0 for a symmetric shape, 2 for an exponential. (0 for a symmetric shape): an exponential has skewness 2, and the mean of n draws has skewness about 2 / sqrt(n).

Reading the code

• np.random.default_rng(42) creates a generator with a fixed seed. Always seed simulations you want others to reproduce.
• size=(5000, 30) makes a table: 5,000 experiments (rows) of 30 draws (columns).
• mean(axis=1) averages across each row.

How large is "large enough"?

There is no single number. The rule of thumb n ≥ 30 works for moderately skewed data. Symmetric data needs far less; heavily skewed data (insurance claims, income, file sizes) can need hundreds. With proportions, a common rule is at least 10 expected successes and 10 expected failures. When in doubt, simulate, exactly as above.

The CLT also fails when the variance is infinite. Some heavy-tailed distributions (such as the Cauchy) have averages that never settle down. These are rare in practice but do appear in some finance and physics data.

Common mistakes

• Applying the CLT to individual values. The CLT is about averages and sums. A single claim size stays skewed no matter how many claims you collect.
• Gambler's fallacy. The LLN does not balance out short runs.
• Confusing SD and SE. The SD describes individual values; the SE describes the uncertainty in the mean. Report the one your reader needs.
• Ignoring dependence. Both theorems assume independent draws. Daily stock returns or time-series sensor readings are correlated, and the effective sample size is smaller than n.

Recap

• LLN: the sample mean converges to the true mean.
• Standard error: σ / sqrt(n); four times the data halves the error.
• CLT: sample means are approximately normal for large n, whatever the original shape.
• Simulate when you are unsure whether n is large enough.

Try it

In the Do tab, simulate thousands of sample means with NumPy and check both theorems numerically.