Describe data, reason with probability, and draw honest conclusions from samples, with short fill-in-the-blank Python exercises.

Statistics Fundamentals

## Why this matters

Every statistical question starts with the same step: look at a column of data and decide what kind of thing it holds. That decision controls everything after it. You can average a column of temperatures, but you cannot average a column of blood types. A chart that works for counts of product categories looks wrong for weights. Analysts, lab technicians and engineers make this call dozens of times a day, usually without noticing. This lesson makes it explicit so you make it correctly.

## Observations and variables

A **dataset** is usually a table. Each **row** is one observation (one patient, one sample, one bridge inspection, one website visit). Each **column** is a **variable**: something you measured or recorded about every observation.

| sample_id | material | batch_grade | tensile_mpa | defects |
|---|---|---|---|---|
| S-01 | steel | A | 412.5 | 0 |
| S-02 | aluminium | B | 276.0 | 2 |
| S-03 | steel | A | 405.1 | 1 |

`sample_id` is a label, not something to analyse. The other four columns are four different kinds of variable.

## Categorical variables

A **categorical** (qualitative) variable puts each observation in a group.

- **Nominal**: groups with no natural order. `material` (steel, aluminium), blood type, country, browser name.
- **Ordinal**: groups with a meaningful order but no fixed distance between them. `batch_grade` (A better than B better than C), survey answers from "strongly disagree" to "strongly agree", pain on a 1 to 5 scale.

Useful summaries for categorical data are **counts**, **proportions** (percent of the total) and the **mode** (the most common group). For ordinal data the **median** also makes sense, because you can line the answers up in order and find the middle one.

A common trap: numbers used as codes. A ZIP code, a phone number, an employee ID or a "region 1, 2, 3" column is categorical even though it is written with digits. The test is simple: would an average mean anything? The average ZIP code is nonsense, so ZIP code is categorical.

## Numerical variables

A **numerical** (quantitative) variable is a measured or counted amount, where arithmetic makes sense.

- **Discrete**: counts that come in whole steps. `defects` (0, 1, 2...), number of support tickets, number of colonies on a plate.
- **Continuous**: measurements that can take any value in a range, limited only by the instrument. `tensile_mpa`, temperature, time, concentration, mass.

For numerical data you can use the **mean**, **median**, **standard deviation**, **range** and percentiles, and charts like histograms and box plots. The next lesson covers those in detail.

### Interval and ratio (a finer split)

Some textbooks split numerical data again:

- **Interval**: differences are meaningful but zero is arbitrary. Temperature in Celsius: 20 °C is not "twice as hot" as 10 °C.
- **Ratio**: there is a true zero, so ratios make sense. Mass, length, Kelvin temperature, money. 20 kg really is twice 10 kg.

You rarely need this distinction in day-to-day work, but it explains why "percent increase" is fine for revenue and misleading for Celsius.

## Choosing a summary: a quick table

| variable type | example | centre | spread | chart |
|---|---|---|---|---|
| nominal | material | mode | counts per group | bar chart |
| ordinal | batch grade | median, mode | range of grades | ordered bar chart |
| discrete | defects | mean, median | standard deviation, range | bar chart or histogram |
| continuous | tensile strength | mean, median | standard deviation, IQR | histogram, box plot |

## Worked example: a clinic visit log

A clinic records `visit_reason` (checkup, injury, illness), `satisfaction` (1 to 5 stars), `wait_minutes` and `prescriptions` (how many were written).

- `visit_reason` is nominal: report "45% checkups, 30% illness, 25% injury".
- `satisfaction` is ordinal: report the median (say 4 stars) and the share of 1 and 2 star ratings. A mean of 3.7 stars is often reported anyway, but be aware it assumes the gap between 1 and 2 stars equals the gap between 4 and 5.
- `wait_minutes` is continuous: report the median wait and the spread.
- `prescriptions` is discrete: the mean (1.4 per visit) is fine, even though no single visit had 1.4.

## How software sees types

Spreadsheets and Python do not know what your data means. A column of ZIP codes is stored as numbers, so the software will happily average it. In Python's `statistics` module, `mean` works on any list of numbers, and `mode` works on any list of hashable values, including text:

```python
import statistics

materials = ["steel", "aluminium", "steel", "steel"]
print(statistics.mode(materials))   # steel

defects = [0, 2, 1, 0, 3]
print(statistics.mean(defects))     # 1.2
```

The software runs whatever you ask. Choosing the right summary is your job.

## Common mistakes

- Averaging codes (IDs, ZIP codes, category numbers).
- Treating ordinal scales as if the steps were equal without saying so.
- Turning a continuous variable into categories ("old" vs "young") too early, which throws information away.
- Forgetting that the same quantity can be recorded at different levels: age in years is numerical, age group "18 to 24" is ordinal.

## Quick recap

- Rows are observations; columns are variables.
- Categorical: nominal (no order) or ordinal (ordered). Summarise with counts, proportions, mode (and median for ordinal).
- Numerical: discrete (counts) or continuous (measurements). Summarise with mean, median, standard deviation and percentiles.
- Ask "would an average mean anything?" before you compute one.

## Try it

Open the Do tab and classify the variables in the quiz.


Classify variables as categorical or numerical and choose the summary that fits each type.

Types of data and variables

Types of Data: Categorical vs Numerical Data

Types of Data: Nominal, Ordinal, Interval/Ratio - Statistics Help

## Why this matters

"The average delivery takes 3 days" and "the average salary is $80,000" sound precise, but they hide two questions: which average, and how much do individual values vary around it? Two production lines can have the same mean part length while one of them scraps a third of its output. Describing **centre** and **spread** together is the most common job in statistics, and choosing the right measures is a skill you will use in every data, lab and engineering role.

## Measures of centre

**Mean**: add the values and divide by how many there are.

```text
data: 4, 8, 6, 5, 7
mean = (4 + 8 + 6 + 5 + 7) / 5 = 30 / 5 = 6
```

**Median**: sort the values and take the middle one. With an even count, average the two middle values.

```text
sorted: 4, 5, 6, 7, 8        median = 6
sorted: 4, 5, 6, 7, 8, 40    median = (6 + 7) / 2 = 6.5
```

**Mode**: the most frequent value. Most useful for categorical data or for discrete data with many repeats.

### Mean vs median: the outlier test

Add one large value and watch what happens. The list `4, 5, 6, 7, 8` has mean 6 and median 6. Add `40`:

- mean = 70 / 6 ≈ 11.7
- median = 6.5

One value dragged the mean almost to double, while the median barely moved. The median is **robust** to outliers; the mean is not. That is why house prices and incomes are usually reported as medians: a few very large values pull the mean far above what a typical person sees.

Use the mean when the data is roughly symmetric with no extreme values (part dimensions, repeated lab measurements). Use the median when the data is skewed or has outliers (wait times, prices, response times). When in doubt, report both; a big gap between them is itself a finding.

## Measures of spread

**Range** = max − min. Simple, but it depends entirely on the two most extreme values.

**Interquartile range (IQR)** = Q3 − Q1, the width of the middle 50% of the data. Robust, like the median. You will use it to flag outliers in the next lesson.

**Variance and standard deviation** measure the typical distance of values from the mean.

1. Subtract the mean from each value (the **deviations**).
2. Square each deviation, so negatives do not cancel positives.
3. Add the squares.
4. Divide by `n − 1` for a **sample** (or by `n` for a whole population). That is the **variance**.
5. Take the square root to get back to the original units. That is the **standard deviation** (SD).

### Worked example

Five measured resistor values in ohms: `98, 101, 100, 99, 102`.

```text
mean = 500 / 5 = 100
deviations:        -2,  1,  0, -1,  2
squared:            4,  1,  0,  1,  4    sum = 10
sample variance = 10 / (5 - 1) = 2.5
sample SD       = sqrt(2.5) ≈ 1.58 ohms
```

A second batch, `90, 110, 100, 95, 105`, also has mean 100, but its squared deviations sum to 250, so the sample SD is sqrt(62.5) ≈ 7.9 ohms. Same centre, very different consistency.

### Why n − 1?

A sample's values sit a little closer to their own mean than to the true population mean, so dividing by `n` slightly underestimates the spread. Dividing by `n − 1` corrects that. In practice: if your data is a sample from something bigger (almost always), use `n − 1`. Spreadsheets call this `STDEV.S`; Python calls it `statistics.stdev`. The population versions are `STDEV.P` and `statistics.pstdev`.

## In Python

The standard library's `statistics` module covers all of this:

```python
import statistics

ohms = [98, 101, 100, 99, 102]
print(statistics.mean(ohms))     # 100
print(statistics.median(ohms))   # 100
print(statistics.stdev(ohms))    # 1.58... (sample, n - 1)
print(statistics.pstdev(ohms))   # 1.41... (population, n)
```

## Common mistakes

- Reporting a mean for skewed data without the median.
- Mixing up sample and population SD (and not saying which one you used).
- Quoting a centre with no spread. "Mean 100 ohms" is half an answer; "mean 100, SD 1.6" is a whole one.
- Forgetting units: the SD is in the same units as the data, the variance is in squared units.

## Quick recap

- Mean: sensitive to outliers. Median: robust. Mode: most common value.
- Range is crude; IQR is robust; SD is the standard measure of spread around the mean.
- Sample SD divides the sum of squared deviations by `n − 1`.
- Always report a centre with a spread, and say which ones.

## Try it

In the Do tab, finish a `summarize` function by writing the sample standard deviation formula.


Compute the mean, median, range and standard deviation, and pick the summary that fits the data.

Mean, median, and spread

Mean, Median, and Mode: Measures of Central Tendency: Crash Course Statistics #3

Measures of Spread: Crash Course Statistics #4

## Why this matters

A mean and a standard deviation compress a whole column into two numbers. That is useful, but it can hide a lot: two peaks, a long tail, a cluster of impossible values from a broken sensor. A **histogram** shows the whole **distribution**, meaning how often each range of values occurs. Looking at it before you summarise is one of the habits that separates careful analysts from careless ones.

## Building a histogram

1. Choose **bins**: equal-width ranges that cover the data, for example 0 to 10, 10 to 20, 20 to 30 minutes.
2. Count how many values fall in each bin.
3. Draw a bar for each bin, with height equal to the count. The bars touch, because the bins are continuous.

A text version of wait times (minutes) at a help desk:

```text
 0-10  ##########   10
10-20  ############ 12
20-30  ######        6
30-40  ###           3
40-50  #             1
50-60                0
60-70  #             1
```

Bin width matters. Too few bins hide structure; too many turn the picture into noise. Start with something like 5 to 15 bins and try a couple of widths.

A histogram is not a bar chart. A bar chart shows categories (blood type, material) with gaps between bars, and the order of bars is up to you. A histogram shows a numerical variable on a number line.

## Describing shape

Describe a distribution with four things: **shape**, **centre**, **spread** and **unusual values**.

**Symmetric**: the left and right halves look roughly alike. Repeated measurements of the same quantity and heights of adults of one sex are classic examples. Mean and median are close.

**Right-skewed** (positively skewed): a long tail to the right. Most values are small, a few are large. Wait times, incomes, file sizes, reaction times, city populations. The mean is pulled toward the tail, so **mean > median**.

**Left-skewed** (negatively skewed): a long tail to the left. Exam scores on an easy test, age at retirement, time to failure for components that rarely fail early. **Mean < median**.

**Bimodal**: two peaks. Often a sign that two groups are mixed together, such as morning and evening shifts, two machines, or two species. Splitting the data by group usually explains it.

The help-desk histogram above is right-skewed: the bulk is under 30 minutes, with a tail out to 60+.

## Quartiles and the box plot

**Quartiles** split sorted data into four equal parts. Q1 is the 25th percentile, Q2 the median, Q3 the 75th percentile. The **IQR** is Q3 − Q1.

A **box plot** draws a box from Q1 to Q3 with a line at the median, and "whiskers" out to the most extreme values that are not outliers. Box plots are compact, which makes them good for comparing several groups side by side.

## Flagging outliers: the 1.5 × IQR rule

A widely used rule (Tukey's fences):

```text
lower fence = Q1 - 1.5 * IQR
upper fence = Q3 + 1.5 * IQR
```

Any value outside the fences is flagged as a possible outlier.

### Worked example

Wait times: `4, 7, 8, 9, 11, 12, 14, 15, 18, 22, 65`. Suppose Q1 = 8 and Q3 = 18 (software may give slightly different quartiles; there are several methods).

```text
IQR = 18 - 8 = 10
lower fence = 8 - 15 = -7
upper fence = 18 + 15 = 33
```

65 is above 33, so it is flagged. Nothing is below −7.

"Flagged" does not mean "delete". An outlier can be a typing error (6.5 entered as 65), a broken instrument, or a real and important event (the one customer who waited an hour). Investigate first. Remove a value only with a documented reason, and say so in your report.

## In Python

```python
import statistics

waits = [4, 7, 8, 9, 11, 12, 14, 15, 18, 22, 65]
q1, q2, q3 = statistics.quantiles(waits, n=4)
print(q1, q2, q3)   # 8.0 12.0 18.0
print(statistics.mean(waits), statistics.median(waits))   # mean 16.8 > median 12: right skew
```

`statistics.quantiles(data, n=4)` returns the three cut points between quartiles.

## Common mistakes

- Summarising with the mean before looking at the shape.
- Using a histogram for categorical data, or a bar chart with gaps for continuous data.
- Deleting outliers automatically because a rule flagged them.
- Judging skew from one bin width only.

## Quick recap

- A histogram shows how often values fall in equal-width bins.
- Describe shape (symmetric, right-skewed, left-skewed, bimodal), centre, spread and unusual values.
- Right skew: mean > median. Left skew: mean < median.
- Fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR flag possible outliers; investigate before removing.

## Try it

In the Do tab, finish an outlier detector by writing the two fence formulas.


Read a histogram's shape, recognise skew, and flag outliers with the 1.5 × IQR rule.

Histograms and distributions

The Shape of Data: Distributions: Crash Course Statistics #7

Skewness - Right, Left & Symmetric Distribution - Mean, Median, & Mode With Boxplots - Statistics

Types of variables, centre and spread, and reading the shape of a distribution with histograms, skew and outliers.