Describe data, reason with probability, and draw honest conclusions from samples, with short fill-in-the-blank Python exercises.

Statistics Fundamentals

## Why this matters

Do taller people weigh more? Does curing time affect concrete strength? Do customers who open more emails buy more? Questions about two numerical variables moving together are everywhere in data, engineering and science. The **correlation coefficient** r summarises the strength and direction of a straight-line relationship in one number. It is also one of the most misused numbers in statistics, so you need to know both how to compute it and what it cannot tell you.

## Start with a scatter plot

Before computing anything, plot the data: one variable on the x-axis, the other on the y-axis, one dot per observation. Look for:

- **Direction**: do points rise to the right (positive) or fall (negative)?
- **Form**: roughly a straight line, a curve, or no pattern?
- **Strength**: tightly clustered around a line, or a loose cloud?
- **Outliers**: points far from the rest.

Correlation only describes **straight-line** (linear) relationships, so the plot is not optional.

## Pearson's r

`r` is always between −1 and +1.

- r = +1: all points exactly on a rising line.
- r = −1: all points exactly on a falling line.
- r = 0: no linear relationship.
- Rough guide for size: |r| below 0.3 weak, 0.3 to 0.7 moderate, above 0.7 strong. Fields differ; in physics 0.9 can be disappointing, in social science 0.3 can be notable.

The formula uses deviations from each mean:

```text
dx = x - mean(x)        dy = y - mean(y)

r = sum(dx * dy) / sqrt( sum(dx^2) * sum(dy^2) )
```

The numerator is large and positive when x and y are above their means together and below their means together. Dividing by the square roots scales the result so that it has no units and stays between −1 and +1.

### Worked example

Hours studied (x) and quiz score (y) for five students:

```text
x: 1, 2, 3, 4, 5        mean 3
y: 52, 60, 61, 70, 77   mean 64

dx:   -2,  -1,  0,  1,  2
dy:  -12,  -4, -3,  6, 13
dx*dy: 24,  4,  0,  6, 26     sum = 60
dx^2:   4,  1,  0,  1,  4     sum = 10
dy^2: 144, 16,  9, 36, 169    sum = 374

r = 60 / sqrt(10 * 374) = 60 / 61.16 ≈ 0.981
```

A strong positive linear relationship in this small sample.

## Properties worth knowing

- r has **no units**, and changing units (inches to cm, °C to °F) does not change it.
- r is **symmetric**: the correlation of x with y equals that of y with x.
- r is **sensitive to outliers**. One extreme point can create a strong r from a cloud, or destroy a real one.
- r measures only **linear** association. Points on a perfect U-shaped curve can have r = 0.
- **r squared** (r²) is the share of the variation in y that a straight line in x accounts for. r = 0.981 gives r² ≈ 0.96. You will meet it again in regression.

## Correlation is not causation

Ice-cream sales and drowning deaths are positively correlated. Ice cream does not cause drowning: hot weather drives both. A hidden third variable like this is called a **confounder** (or lurking variable). Other reasons two variables correlate without one causing the other:

- **Reverse causation**: cities with more police officers have more crime, partly because high crime leads cities to hire police.
- **Coincidence**: search enough unrelated time series and some will match closely by chance.
- **Selection effects**: studying only one group can create a relationship that does not exist in the whole population.

To show cause, you generally need a **randomised controlled experiment** (randomly assign the treatment), or very careful study design and domain knowledge. Correlation is a reason to investigate, not a conclusion.

## In Python

```python
import statistics

hours = [1, 2, 3, 4, 5]
score = [52, 60, 61, 70, 77]
print(statistics.correlation(hours, score))   # 0.981...
```

`statistics.correlation` (Python 3.10 and later) computes Pearson's r.

## Common mistakes

- Computing r without plotting the data.
- Saying "no relationship" when r ≈ 0 but the plot shows a curve.
- Treating correlation as proof of cause.
- Comparing r values from samples with very different ranges of x (a narrow range weakens r).

## Quick recap

- Always plot first: direction, form, strength, outliers.
- r ranges from −1 to +1 and measures linear association only.
- r = sum(dx·dy) / sqrt(sum(dx²)·sum(dy²)).
- Correlation is not causation: look for confounders and reverse causation.

## Try it

In the Do tab, write the final line of the Pearson's r formula.


Compute Pearson's correlation coefficient r and interpret its sign, strength and limits.

Correlation

Correlation Doesn't Equal Causation: Crash Course Statistics #8

Calculating correlation coefficient r | AP Statistics | Khan Academy

## Why this matters

Correlation tells you how strongly two variables move together. **Regression** goes a step further: it gives you a line you can use to predict one variable from the other, and a slope that says how much y changes per unit of x. Engineers use it for calibration curves, lab technicians for standard curves, analysts for "every extra $1,000 of ad spend brings about 30 more sign-ups". Simple linear regression, with one predictor and a straight line, is the foundation for every more advanced model you will meet.

## The model

```text
predicted y = intercept + slope * x
```

- **x** is the explanatory (predictor) variable.
- **y** is the response (outcome) variable.
- **slope** (b): the change in predicted y for a one-unit increase in x.
- **intercept** (a): the predicted y when x = 0.

Unlike correlation, regression is not symmetric. Predicting y from x gives a different line from predicting x from y, so decide which variable is the response.

## Least squares

Many lines could pass through a cloud of points. The **least-squares** line is the one that makes the sum of squared **residuals** as small as possible. A residual is the vertical gap between a point and the line:

```text
residual = actual y - predicted y
```

Squaring keeps positive and negative gaps from cancelling and penalises big misses most. The formulas:

```text
dx = x - mean(x)       dy = y - mean(y)

slope     = sum(dx * dy) / sum(dx^2)
intercept = mean(y) - slope * mean(x)
```

The line always passes through the point (mean x, mean y).

### Worked example: a calibration curve

A lab measures the absorbance of standard solutions of known concentration (mg/L):

```text
conc (x):        0,    2,    4,    6,    8      mean 4
absorbance (y):  0.02, 0.21, 0.39, 0.62, 0.81   mean 0.41

dx:     -4,    -2,    0,    2,    4
dy:  -0.39, -0.20, -0.02, 0.21, 0.40
dx*dy: 1.56, 0.40,  0.00, 0.42, 1.60     sum = 3.98
dx^2:  16,   4,     0,    4,    16       sum = 40

slope     = 3.98 / 40 = 0.0995 absorbance units per mg/L
intercept = 0.41 - 0.0995 * 4 = 0.012
```

The fitted line is `absorbance ≈ 0.012 + 0.0995 × conc`. An unknown sample reads 0.50, so its concentration is about (0.50 − 0.012) / 0.0995 ≈ 4.9 mg/L. Real calibration work follows the lab's validated method and acceptance criteria; this shows only the arithmetic.

## Interpreting the line

- **Slope**: "Each additional mg/L raises predicted absorbance by about 0.0995." Always state units, and say "predicted" or "on average".
- **Intercept**: only meaningful if x = 0 makes sense and is inside the data. Here 0 mg/L is a blank, so 0.012 is the blank's predicted reading. For "salary vs years of experience", x = 0 is a new hire and is reasonable; for "weight vs height", height 0 is nonsense.
- **r²**: the share of variation in y explained by the line. Here r² is about 0.999, typical for a good calibration. For people or markets, 0.3 can be useful.

## Check the residuals

Plot residuals against x. A good linear fit leaves a shapeless band around zero. A curve in the residuals means the relationship is not linear. A funnel (residuals spreading out as x grows) means the variability changes, and the simple model's uncertainty estimates will be off.

## Extrapolation

The line is trustworthy only inside the range of x you measured. The calibration above used 0 to 8 mg/L. Predicting the absorbance at 40 mg/L assumes the line stays straight five times beyond the data, and instruments often saturate. Predicting outside the data range is called **extrapolation** and is a common source of embarrassing forecasts.

## In Python

```python
import statistics

conc = [0, 2, 4, 6, 8]
absorb = [0.02, 0.21, 0.39, 0.62, 0.81]
fit = statistics.linear_regression(conc, absorb)
print(fit.slope, fit.intercept)   # 0.0995 0.012
```

`statistics.linear_regression` (Python 3.10 and later) returns the least-squares slope and intercept. Libraries such as statsmodels and scikit-learn add standard errors, multiple predictors and diagnostics; later courses use them.

## Common mistakes

- Interpreting an intercept where x = 0 is meaningless.
- Extrapolating far beyond the data.
- Reading the slope as a causal effect from observational data.
- Fitting a line without looking at the scatter plot and residuals.

## Quick recap

- predicted y = intercept + slope × x, fitted by least squares.
- slope = sum(dx·dy) / sum(dx²); intercept = mean(y) − slope × mean(x).
- Interpret the slope with units; interpret the intercept only when x = 0 is meaningful.
- Check residuals, and do not extrapolate.

## Try it

In the Do tab, write the slope formula and use the line to read an unknown sample.


Fit a least-squares line, interpret its slope and intercept, and use it to predict responsibly.

Simple linear regression

Regression: Crash Course Statistics #32

Linear Regression Using Least Squares Method - Line of Best Fit Equation

## Why this matters

Most misleading statistics are not lies. Every number can be correct and the conclusion can still be wrong, because of how the data was grouped, charted or described. As a data analyst, scientist or engineer you will both read and produce these numbers. Being able to say "that chart's axis starts at 95" or "that comparison ignores group size" in a meeting is one of the most valuable skills this course teaches.

## Simpson's paradox

A trend that appears in every group can reverse when the groups are combined. The classic shape: two treatments for kidney stones.

```text
                 Treatment A       Treatment B
small stones     81/87  = 93%      234/270 = 87%
large stones     192/263 = 73%     55/80   = 69%
all patients     273/350 = 78%     289/350 = 83%
```

A has the higher success rate for small stones **and** for large stones, yet B looks better overall. How? Doctors gave A mostly to the harder large-stone cases (263 of A's 350) and B mostly to easy small-stone cases (270 of B's 350). Stone size is a **confounder**: it affects both which treatment was given and the chance of success. The combined numbers mostly compare case mix, not treatments.

The lesson: when groups differ in size or composition, look at the breakdown, and ask what variable might drive both the "treatment" and the outcome. The same thing happens with admission rates across departments, hospital death rates (good hospitals take the sickest patients), and A/B tests where traffic sources shift during the test.

## Misleading charts

**Truncated axis.** A bar chart of conversion rates of 96.1% vs 96.8% with the y-axis starting at 95% makes a 0.7-point difference look like a doubling. Bar charts should start at zero, because bar length is the message. Line charts may zoom in to show change, but the axis should be labelled clearly.

**Inconsistent scales.** Two charts side by side with different y-axis ranges invite a false comparison. So does an x-axis with uneven time steps (2015, 2020, 2021, 2022).

**Area and 3-D tricks.** Doubling both the width and height of an icon to show "twice as much" makes it look four times as big. 3-D pie charts distort the slices nearest the viewer.

**Cherry-picked windows.** Starting a stock chart at a low point or ending a temperature chart at an unusual year can show almost any trend you like. Ask why the chart starts and ends where it does.

**Dual axes.** Two lines on different y-axes can be made to cross or move together by rescaling either axis.

## Misleading numbers

**Relative vs absolute risk.** "Doubles the risk" sounds alarming. If the risk goes from 1 in 10,000 to 2 in 10,000, the absolute increase is 0.01 percentage points. Report both.

**Percent of what?** "Sales up 50%!" from 2 units to 3 means little. A small base makes percent changes huge.

**Averages hiding spread.** "Average response time is 200 ms" can hide 5% of requests taking 4 seconds. Report medians, percentiles (p95, p99) and spread.

**Survivorship and selection.** "Our graduates earn $90,000" may count only graduates who answered the survey or who completed the program.

**Correlation presented as cause.** "People who drink coffee live longer" from observational data does not show that coffee extends life. Look for confounders such as income or overall health.

**Statistical vs practical significance.** With a million users, a 0.1% improvement can have a tiny p-value and still not be worth building.

**Regression to the mean.** Extreme results tend to be followed by more ordinary ones. A team that hires a new coach after its worst season will probably improve next season regardless. Without a comparison group, you cannot credit the coach.

## A checklist for any claim

1. Who collected the data, and how was the sample chosen?
2. Compared with what? Is there a baseline or control group?
3. Absolute numbers or only percentages? What is the base?
4. Could a third variable explain it? Look at the breakdown by group.
5. Does the chart's axis start at zero (for bars)? Are scales consistent?
6. Is the difference large enough to matter, not just "significant"?
7. Why does the time window start and end where it does?

## Quick recap

- Simpson's paradox: a confounder plus unequal group mix can reverse a trend when groups are combined.
- Charts mislead through truncated axes, inconsistent scales, area effects, cherry-picked windows and dual axes.
- Numbers mislead through relative-only risk, small bases, averages without spread, selection, causal claims from correlation and regression to the mean.
- Ask: compared with what, out of how many, and what else could explain it?

## Try it

Open the Do tab and test the checklist on the quiz scenarios.


Spot Simpson's paradox, misleading charts and other ways true numbers tell false stories.

Misleading statistics

Simpson's Paradox

How to spot a misleading graph - Lea Gaslowitz

Correlation, simple linear regression, and how Simpson's paradox and misleading charts distort conclusions.