Sampling and bias

Why this matters

You almost never measure everything. A lab tests a few vials from a batch, a pollster calls a thousand people out of millions, an engineer cores a handful of concrete samples from a whole slab. Statistics lets you reason from that sample to the population, but only if the sample was chosen well. No formula can rescue a biased sample. Spotting bias in a study design is a skill every analyst, scientist and engineer needs, and it needs no math at all.

Population, sample, parameter, statistic

• Population: everything you want to draw conclusions about (all customers, all parts made this month, all adults in a region).
• Sample: the subset you actually measure.
• Parameter: a number describing the population, usually unknown (the true mean, the true defect rate).
• Statistic: the matching number computed from the sample (the sample mean, the sample defect rate). You use statistics to estimate parameters.

Two kinds of error separate a statistic from its parameter:

• Random sampling error: different random samples give slightly different answers. It shrinks as the sample gets bigger, and the next lessons show how to measure it.
• Bias: a systematic push in one direction. It does not shrink with sample size. A biased sample of a million is still biased.

Good sampling methods

Simple random sample (SRS): every member of the population has the same chance of being picked, and every group of size n is equally likely. Draw IDs with a random number generator, not "whoever is nearby".

Stratified sample: split the population into groups (strata) that matter, such as regions or shifts, then take a random sample within each. Guarantees every group is represented and often gives more precise estimates.

Cluster sample: split the population into natural clusters (schools, pallets, city blocks), randomly pick some clusters, and measure everything in them. Cheaper, but less precise if clusters differ a lot.

Systematic sample: pick every k-th item from a list or production line, starting at a random point. Simple and usually fine, but dangerous if the process has a cycle that lines up with k (for example, every 10th part comes from the same mould cavity).

Common sources of bias

Convenience sampling: measuring whoever or whatever is easy to reach. Surveying customers in the store at 10 a.m. on a weekday misses everyone who works days.

Voluntary response: people choose whether to respond. Online reviews and call-in polls over-represent people with strong (usually negative) opinions.

Undercoverage: part of the population cannot be selected at all. A phone survey of landlines misses most young adults.

Nonresponse: people are selected but do not answer, and non-responders differ from responders. A satisfaction survey that busy or unhappy customers ignore looks too rosy.

Survivorship bias: only the "survivors" are visible. Studying successful startups to find what makes companies succeed ignores the failed ones that did the same things. Inspecting only the returned products tells you nothing about the ones customers threw away.

Response bias: the way a question is asked pushes the answer. "Don't you agree that the new policy is unfair?" is a leading question. People also under-report things that feel embarrassing.

Measurement bias: the instrument or process reads systematically high or low. A scale that is not zeroed, a thermometer placed in the sun.

Worked example: a study design

A hospital wants the average wait time in its emergency department. A student stands in the waiting room on Tuesday afternoons for a month and asks people leaving how long they waited.

Problems:

1. Convenience and time coverage: Tuesday afternoons are not representative; nights and weekends differ.
2. Recall bias: patients estimate their own wait, and long waits feel longer.
3. Survivorship: people who gave up and left before being seen are never asked.

A better design: pull the recorded arrival and triage timestamps for a random sample of visits across all days and hours, stratified by shift.

Sample size versus bias

A famous lesson comes from the 1936 US presidential election. A magazine mailed millions of ballots to people drawn from phone books and car registrations, which at the time over-represented wealthier voters, and only a fraction mailed them back. It predicted a landslide for the losing candidate. A much smaller but better-designed poll got it right. Size does not fix bias.

Common mistakes

• Believing a big sample is automatically a good one.
• Calling a convenience sample "random" because nobody chose who to include on purpose.
• Ignoring who did not respond.
• Generalising beyond the population that was actually sampled.

Quick recap

• Statistics (from samples) estimate parameters (of populations).
• Random error shrinks with sample size; bias does not.
• Prefer random, stratified, cluster or carefully started systematic samples.
• Watch for convenience, voluntary response, undercoverage, nonresponse, survivorship, response and measurement bias.

Try it

Open the Do tab and find the bias in each study design.