Correlation

Why this matters

Do taller people weigh more? Does curing time affect concrete strength? Do customers who open more emails buy more? Questions about two numerical variables moving together are everywhere in data, engineering and science. The correlation coefficientPearson's r, a number from -1 to +1 giving the direction and strength of a straight-line relationship. r summarises the strength and direction of a straight-line relationship in one number. It is also one of the most misused numbers in statistics, so you need to know both how to compute it and what it cannot tell you.

Start with a scatter plot

Before computing anything, plot the data: one variable on the x-axis, the other on the y-axis, one dot per observation. Look for:

• Direction: do points rise to the right (positive) or fall (negative)?
• Form: roughly a straight line, a curve, or no pattern?
• Strength: tightly clustered around a line, or a loose cloud?
• Outliers: points far from the rest.

Correlation only describes straight-line (linear) relationships, so the plot is not optional.

Pearson's r

r is always between −1 and +1.

• r = +1: all points exactly on a rising line.
• r = −1: all points exactly on a falling line.
• r = 0: no linear relationship.
• Rough guide for size: |r| below 0.3 weak, 0.3 to 0.7 moderate, above 0.7 strong. Fields differ; in physics 0.9 can be disappointing, in social science 0.3 can be notable.

The formula uses deviations from each mean:

dx = x - mean(x)        dy = y - mean(y)

r = sum(dx * dy) / sqrt( sum(dx^2) * sum(dy^2) )

The numerator is large and positive when x and y are above their means together and below their means together. Dividing by the square roots scales the result so that it has no units and stays between −1 and +1.

Worked example

Hours studied (x) and quiz score (y) for five students:

x: 1, 2, 3, 4, 5        mean 3
y: 52, 60, 61, 70, 77   mean 64

dx:   -2,  -1,  0,  1,  2
dy:  -12,  -4, -3,  6, 13
dx*dy: 24,  4,  0,  6, 26     sum = 60
dx^2:   4,  1,  0,  1,  4     sum = 10
dy^2: 144, 16,  9, 36, 169    sum = 374

r = 60 / sqrt(10 * 374) = 60 / 61.16 ≈ 0.981

A strong positive linear relationship in this small sample.

Properties worth knowing

• r has no units, and changing units (inches to cm, °C to °F) does not change it.
• r is symmetric: the correlation of x with y equals that of y with x.
• r is sensitive to outliers. One extreme point can create a strong r from a cloud, or destroy a real one.
• r measures only linear association. Points on a perfect U-shaped curve can have r = 0.
• r squaredThe share of the variation in y that a straight line in x accounts for. (r²) is the share of the variation in y that a straight line in x accounts for. r = 0.981 gives r² ≈ 0.96. You will meet it again in regression.

Correlation is not causation

Ice-cream sales and drowning deaths are positively correlated. Ice cream does not cause drowning: hot weather drives both. A hidden third variable like this is called a confounderA hidden third variable that drives two others, making them correlate without one causing the other. (or lurking variable). Other reasons two variables correlate without one causing the other:

• Reverse causationWhen the supposed effect actually causes the supposed cause.: cities with more police officers have more crime, partly because high crime leads cities to hire police.
• Coincidence: search enough unrelated time series and some will match closely by chance.
• Selection effects: studying only one group can create a relationship that does not exist in the whole population.

To show cause, you generally need a randomised controlled experimentA study that randomly assigns a treatment, which is the usual way to show cause and effect. (randomly assign the treatment), or very careful study design and domain knowledge. Correlation is a reason to investigate, not a conclusion.

In Python

import statistics

hours = [1, 2, 3, 4, 5]
score = [52, 60, 61, 70, 77]
print(statistics.correlation(hours, score))   # 0.981...

statistics.correlation (Python 3.10 and later) computes Pearson's r.

Common mistakes

• Computing r without plotting the data.
• Saying "no relationship" when r ≈ 0 but the plot shows a curve.
• Treating correlation as proof of cause.
• Comparing r values from samples with very different ranges of x (a narrow range weakens r).

Quick recap

• Always plot first: direction, form, strength, outliers.
• r ranges from −1 to +1 and measures linear association only.
• r = sum(dx·dy) / sqrt(sum(dx²)·sum(dy²)).
• Correlation is not causation: look for confounders and reverse causation.

Try it

In the Do tab, write the final line of the Pearson's r formula.