Describe data, reason with probability, and draw honest conclusions from samples, with short fill-in-the-blank Python exercises.

Statistics Fundamentals

## Why this matters

You almost never measure everything. A lab tests a few vials from a batch, a pollster calls a thousand people out of millions, an engineer cores a handful of concrete samples from a whole slab. Statistics lets you reason from that **sample** to the **population**, but only if the sample was chosen well. No formula can rescue a biased sample. Spotting bias in a study design is a skill every analyst, scientist and engineer needs, and it needs no math at all.

## Population, sample, parameter, statistic

- **Population**: everything you want to draw conclusions about (all customers, all parts made this month, all adults in a region).
- **Sample**: the subset you actually measure.
- **Parameter**: a number describing the population, usually unknown (the true mean, the true defect rate).
- **Statistic**: the matching number computed from the sample (the sample mean, the sample defect rate). You use statistics to estimate parameters.

Two kinds of error separate a statistic from its parameter:

- **Random sampling error**: different random samples give slightly different answers. It shrinks as the sample gets bigger, and the next lessons show how to measure it.
- **Bias**: a systematic push in one direction. It does **not** shrink with sample size. A biased sample of a million is still biased.

## Good sampling methods

**Simple random sample (SRS)**: every member of the population has the same chance of being picked, and every group of size n is equally likely. Draw IDs with a random number generator, not "whoever is nearby".

**Stratified sample**: split the population into groups (strata) that matter, such as regions or shifts, then take a random sample within each. Guarantees every group is represented and often gives more precise estimates.

**Cluster sample**: split the population into natural clusters (schools, pallets, city blocks), randomly pick some clusters, and measure everything in them. Cheaper, but less precise if clusters differ a lot.

**Systematic sample**: pick every k-th item from a list or production line, starting at a random point. Simple and usually fine, but dangerous if the process has a cycle that lines up with k (for example, every 10th part comes from the same mould cavity).

## Common sources of bias

**Convenience sampling**: measuring whoever or whatever is easy to reach. Surveying customers in the store at 10 a.m. on a weekday misses everyone who works days.

**Voluntary response**: people choose whether to respond. Online reviews and call-in polls over-represent people with strong (usually negative) opinions.

**Undercoverage**: part of the population cannot be selected at all. A phone survey of landlines misses most young adults.

**Nonresponse**: people are selected but do not answer, and non-responders differ from responders. A satisfaction survey that busy or unhappy customers ignore looks too rosy.

**Survivorship bias**: only the "survivors" are visible. Studying successful startups to find what makes companies succeed ignores the failed ones that did the same things. Inspecting only the returned products tells you nothing about the ones customers threw away.

**Response bias**: the way a question is asked pushes the answer. "Don't you agree that the new policy is unfair?" is a leading question. People also under-report things that feel embarrassing.

**Measurement bias**: the instrument or process reads systematically high or low. A scale that is not zeroed, a thermometer placed in the sun.

## Worked example: a study design

A hospital wants the average wait time in its emergency department. A student stands in the waiting room on Tuesday afternoons for a month and asks people leaving how long they waited.

Problems:

1. **Convenience and time coverage**: Tuesday afternoons are not representative; nights and weekends differ.
2. **Recall bias**: patients estimate their own wait, and long waits feel longer.
3. **Survivorship**: people who gave up and left before being seen are never asked.

A better design: pull the recorded arrival and triage timestamps for a random sample of visits across all days and hours, stratified by shift.

## Sample size versus bias

A famous lesson comes from the 1936 US presidential election. A magazine mailed millions of ballots to people drawn from phone books and car registrations, which at the time over-represented wealthier voters, and only a fraction mailed them back. It predicted a landslide for the losing candidate. A much smaller but better-designed poll got it right. **Size does not fix bias.**

## Common mistakes

- Believing a big sample is automatically a good one.
- Calling a convenience sample "random" because nobody chose who to include on purpose.
- Ignoring who did not respond.
- Generalising beyond the population that was actually sampled.

## Quick recap

- Statistics (from samples) estimate parameters (of populations).
- Random error shrinks with sample size; bias does not.
- Prefer random, stratified, cluster or carefully started systematic samples.
- Watch for convenience, voluntary response, undercoverage, nonresponse, survivorship, response and measurement bias.

## Try it

Open the Do tab and find the bias in each study design.


Recognise common sampling methods and spot the biases that make a sample misrepresent its population.

Sampling and bias

Sampling Methods and Bias with Surveys: Crash Course Statistics #10

Techniques for random sampling and avoiding bias | Study design | AP Statistics | Khan Academy

## Why this matters

A sample mean is a single best guess, and it will almost never equal the true population mean exactly. A **confidence interval** adds an honest range: "the mean fill volume is 501.2 mL, give or take 0.8 mL". Reports, lab certificates, polls ("±3 percentage points") and engineering test results all use intervals. Knowing how to build one, and how to read one without overclaiming, is core statistical literacy.

## The idea: estimate ± margin of error

```text
confidence interval = estimate ± margin of error
```

The margin of error depends on two things: how much the estimate would vary from sample to sample (its **standard error**), and how confident you want to be (a **critical value**).

## Standard error of the mean

If you took many random samples of size n and computed each mean, those means would themselves form a distribution. Its spread is the **standard error (SE)**:

```text
SE = sd / sqrt(n)
```

where `sd` is the sample standard deviation. Two things to notice:

- More variable data (bigger sd) gives a bigger SE.
- Bigger samples give a smaller SE, but only with the square root: to halve the SE you need **four times** the data.

A key fact, the **central limit theorem**, says the distribution of sample means is approximately normal for reasonably large n (a common rule of thumb is 30 or more), even when the data itself is not normal. That is why the normal curve from the last module shows up here.

## The 95% interval

For a normal distribution, 95% of the area lies within 1.96 SDs of the centre. So:

```text
95% CI = mean ± 1.96 * sd / sqrt(n)
```

Other levels use other critical values: 1.645 for 90%, 2.576 for 99%. Higher confidence means a wider interval.

### Worked example

A bottling line samples 36 bottles. Mean fill = 501.2 mL, sample SD = 2.4 mL.

```text
SE     = 2.4 / sqrt(36) = 2.4 / 6 = 0.4 mL
margin = 1.96 * 0.4 = 0.784 mL
95% CI = 501.2 ± 0.78  ->  (500.42, 501.98) mL
```

The whole interval is above 500 mL, so the data supports the claim that the line fills at least the labelled 500 mL on average.

### Small samples and the t distribution

When n is small, using 1.96 makes the interval too narrow, because the sample SD is itself an uncertain estimate. The fix is the **t distribution**, which has fatter tails and a critical value that depends on n − 1 ("degrees of freedom"). For n = 10, the 95% value is about 2.26 instead of 1.96. Spreadsheets (`T.INV.2T`) and libraries such as SciPy provide it; the later course Probability & Statistical Inference uses it in depth. For n of 30 or more the difference is small, which is why this lesson uses 1.96.

## What "95% confident" means

This is the part people get wrong. The true mean is a fixed number; it is either in your interval or not. "95%" describes the **method**: if you repeated the whole sampling process many times and built an interval each time, about 95% of those intervals would contain the true mean.

Correct: "We are 95% confident the true mean fill volume is between 500.4 and 502.0 mL."

Incorrect:

- "95% of bottles are between 500.4 and 502.0 mL." The interval is about the **mean**, not individual bottles. Individual fills vary much more (SD 2.4 mL).
- "There is a 95% chance the true mean is in this particular interval." Close, and common in casual speech, but strictly the probability statement is about the procedure.

## Margin of error for proportions

For a proportion (share of yes answers, defect rate) the same pattern applies:

```text
SE = sqrt(p * (1 - p) / n)
```

A poll of 1,000 people with p = 0.5 has SE ≈ 0.0158 and margin 1.96 × 0.0158 ≈ 0.031: the familiar "±3 points".

## Common mistakes

- Reading the interval as the range of individual values.
- Forgetting that the margin covers only random sampling error. A biased sample can produce a narrow, confident, wrong interval.
- Using 1.96 with very small samples instead of a t value.
- Using the SD where the SE belongs (forgetting to divide by sqrt(n)).

## Quick recap

- CI = estimate ± critical value × standard error.
- SE of a mean = sd / sqrt(n); four times the data halves it.
- 95% uses 1.96 for large samples; small samples need t.
- "95% confident" describes how often the method captures the true value.

## Try it

In the Do tab, write the margin of error formula and build an interval for the bottling data.


Compute a 95% confidence interval for a mean and interpret it correctly.

Confidence intervals

Confidence Intervals: Crash Course Statistics #20

Confidence intervals and margin of error | AP Statistics | Khan Academy

## Why this matters

Did the new website layout really raise sign-ups, or did we get a lucky week? Is this coin, die or production process behaving differently from what we expect? A **hypothesis test** is a standard procedure for asking whether data is surprising under a "nothing special is going on" assumption. You will see p-values in A/B tests, clinical studies, lab method validation and quality reports, and you will need to read them without being fooled.

## The logic in four steps

1. **State two hypotheses.**
   - The **null hypothesis** H0: no effect, no difference, the default. "The coin is fair: p = 0.5."
   - The **alternative** H1: what you suspect. "The coin is not fair: p ≠ 0.5."
2. **Choose a significance level** α before looking at the data. α = 0.05 is common.
3. **Compute a test statistic** that measures how far the data is from what H0 predicts, in standard-error units.
4. **Find the p-value**: the probability, **assuming H0 is true**, of getting a result at least as extreme as the one observed. If p ≤ α, reject H0. Otherwise, fail to reject it.

It works like a trial: H0 is "innocent". You only convict when the evidence would be very surprising if the defendant were innocent.

## A z-test for a proportion

For a yes/no outcome with n trials, x successes and a null value p0:

```text
p_hat = x / n
SE    = sqrt(p0 * (1 - p0) / n)      (the SE under H0)
z     = (p_hat - p0) / SE
two-sided p-value = 2 * (1 - cdf(|z|))
```

This uses the normal approximation, which is reasonable when `n * p0` and `n * (1 − p0)` are both at least about 10.

### Worked example: is the coin fair?

You flip a coin 100 times and get 60 heads.

```text
p_hat = 60 / 100 = 0.60
SE    = sqrt(0.5 * 0.5 / 100) = 0.05
z     = (0.60 - 0.50) / 0.05 = 2.0
p     = 2 * (1 - cdf(2.0)) = 2 * (1 - 0.9772) = 0.0455
```

p = 0.046 ≤ 0.05, so at α = 0.05 you reject H0: 60 heads out of 100 would be fairly surprising from a fair coin. With 56 heads, z = 1.2 and p ≈ 0.23: not surprising, so you fail to reject H0.

### Worked example: A/B sign-ups

A sign-up page historically converts 10% of visitors. With a new design, 1,000 visitors produce 124 sign-ups.

```text
p_hat = 0.124
SE    = sqrt(0.10 * 0.90 / 1000) ≈ 0.00949
z     = 0.024 / 0.00949 ≈ 2.53
p     ≈ 0.011
```

Strong evidence the conversion rate is no longer 10%. The size of the change (about 2.4 percentage points) is a separate question from whether it is real.

## Reading p-values correctly

A p-value of 0.03 means: **if H0 were true**, results this extreme would happen about 3% of the time.

It does **not** mean:

- "There is a 3% chance H0 is true." The p-value assumes H0; it cannot also give its probability.
- "There is a 97% chance the effect is real."
- "The effect is large or important." With huge samples, tiny and useless effects become "significant". Always report the effect size, ideally with a confidence interval.

"Fail to reject H0" is not "H0 is proven". It means the data was not surprising enough; the effect may exist but be too small for this sample to detect.

## Two kinds of error

- **Type I error** (false positive): rejecting a true H0. Its rate is α.
- **Type II error** (false negative): failing to reject a false H0. Bigger samples reduce it; this is the **power** of a test.

Lowering α to 0.01 cuts false positives but makes real effects harder to detect. The right balance depends on the cost of each mistake.

## p-hacking

If you test 20 unrelated things at α = 0.05, you expect about one "significant" result by pure chance. Testing many variables, subgroups or stopping points and reporting only the ones that cross 0.05 is called **p-hacking**, and it produces findings that do not replicate. Decide on the hypothesis, the test and the sample size before you look.

## Common mistakes

- Treating p as the probability that H0 is true.
- Confusing "statistically significant" with "important".
- Saying "we proved H0" after a non-significant result.
- Running many tests and reporting only the winners.

## Quick recap

- H0 is the default; the p-value is P(data this extreme | H0).
- Reject H0 when p ≤ α (chosen in advance).
- z-test for a proportion: z = (p̂ − p0) / sqrt(p0(1 − p0)/n).
- Report the effect size, not just the p-value; beware multiple testing.

## Try it

In the Do tab, write the z statistic and test the coin and the sign-up page.


Set up null and alternative hypotheses, run a simple z-test, and interpret the p-value without overclaiming.

Hypothesis tests and p-values

How P-Values Help Us Test Hypotheses: Crash Course Statistics #21

p-values: What they are and how to interpret them

Sampling methods and bias, confidence intervals, and hypothesis tests with p-values.