Simple linear regression

Why this matters

Correlation tells you how strongly two variables move together. Regression goes a step further: it gives you a line you can use to predict one variable from the other, and a slopeThe change in predicted y for a one-unit increase in x. that says how much y changes per unit of x. Engineers use it for calibration curves, lab technicians for standard curves, analysts for "every extra $1,000 of ad spend brings about 30 more sign-ups". Simple linear regression, with one predictor and a straight line, is the foundation for every more advanced model you will meet.

The model

predicted y = intercept + slope * x

• x is the explanatory (predictor) variable.
• y is the response (outcome) variable.
• slope (b): the change in predicted y for a one-unit increase in x.
• interceptThe predicted y when x is 0, meaningful only if x = 0 makes sense and is within the data. (a): the predicted y when x = 0.

Unlike correlation, regression is not symmetric. Predicting y from x gives a different line from predicting x from y, so decide which variable is the response.

Least squares

Many lines could pass through a cloud of points. The least-squaresThe line that makes the sum of squared residuals as small as possible. line is the one that makes the sum of squared residuals as small as possible. A residualThe vertical gap between a point and the line: actual y minus predicted y. is the vertical gap between a point and the line:

residual = actual y - predicted y

Squaring keeps positive and negative gaps from cancelling and penalises big misses most. The formulas:

dx = x - mean(x)       dy = y - mean(y)

slope     = sum(dx * dy) / sum(dx^2)
intercept = mean(y) - slope * mean(x)

The line always passes through the point (mean x, mean y).

Worked example: a calibration curve

A lab measures the absorbance of standard solutions of known concentration (mg/L):

conc (x):        0,    2,    4,    6,    8      mean 4
absorbance (y):  0.02, 0.21, 0.39, 0.62, 0.81   mean 0.41

dx:     -4,    -2,    0,    2,    4
dy:  -0.39, -0.20, -0.02, 0.21, 0.40
dx*dy: 1.56, 0.40,  0.00, 0.42, 1.60     sum = 3.98
dx^2:  16,   4,     0,    4,    16       sum = 40

slope     = 3.98 / 40 = 0.0995 absorbance units per mg/L
intercept = 0.41 - 0.0995 * 4 = 0.012

The fitted line is absorbance ≈ 0.012 + 0.0995 × conc. An unknown sample reads 0.50, so its concentration is about (0.50 − 0.012) / 0.0995 ≈ 4.9 mg/L. Real calibration work follows the lab's validated method and acceptance criteria; this shows only the arithmetic.

Interpreting the line

• Slope: "Each additional mg/L raises predicted absorbance by about 0.0995." Always state units, and say "predicted" or "on average".
• Intercept: only meaningful if x = 0 makes sense and is inside the data. Here 0 mg/L is a blank, so 0.012 is the blank's predicted reading. For "salary vs years of experience", x = 0 is a new hire and is reasonable; for "weight vs height", height 0 is nonsense.
• r²: the share of variation in y explained by the line. Here r² is about 0.999, typical for a good calibration. For people or markets, 0.3 can be useful.

Check the residuals

Plot residuals against x. A good linear fit leaves a shapeless band around zero. A curve in the residuals means the relationship is not linear. A funnel (residuals spreading out as x grows) means the variability changes, and the simple model's uncertainty estimates will be off.

Extrapolation

The line is trustworthy only inside the range of x you measured. The calibration above used 0 to 8 mg/L. Predicting the absorbance at 40 mg/L assumes the line stays straight five times beyond the data, and instruments often saturate. Predicting outside the data range is called extrapolationUsing a fitted line to predict outside the range of x values that were measured. and is a common source of embarrassing forecasts.

In Python

import statistics

conc = [0, 2, 4, 6, 8]
absorb = [0.02, 0.21, 0.39, 0.62, 0.81]
fit = statistics.linear_regression(conc, absorb)
print(fit.slope, fit.intercept)   # 0.0995 0.012

statistics.linear_regression (Python 3.10 and later) returns the least-squares slope and intercept. Libraries such as statsmodels and scikit-learn add standard errors, multiple predictors and diagnostics; later courses use them.

Common mistakes

• Interpreting an intercept where x = 0 is meaningless.
• Extrapolating far beyond the data.
• Reading the slope as a causal effect from observational data.
• Fitting a line without looking at the scatter plot and residuals.

Quick recap

• predicted y = intercept + slope × x, fitted by least squares.
• slope = sum(dx·dy) / sum(dx²); intercept = mean(y) − slope × mean(x).
• Interpret the slope with units; interpret the intercept only when x = 0 is meaningful.
• Check residuals, and do not extrapolate.

Try it

In the Do tab, write the slope formula and use the line to read an unknown sample.