Multiple regression

Why this matters

Almost every quantitative career this course serves asks the same question in different clothes: how does an outcome change when several inputs change at once? A chemist wants to know how yield depends on temperature and catalyst load. A materials scientist links hardness to carbon content and cooling rate. An operations research analyst relates delivery time to distance, traffic and driver experience. Multiple linear regression is the workhorse for all of them, and the ideas in this module (interpretation, diagnostics, generalised models) carry straight into the design of experiments later in the course.

The model

With one predictor you fit a line. With p predictors you fit a flat surface (a hyperplane):

y = b0 + b1*x1 + b2*x2 + ... + bp*xp + error

The coefficients are chosen by ordinary least squares (OLS): they minimise the sum of squared residuals, SSE = sum (y - fitted)^2. In matrix form, stack a column of ones (for the intercept) next to the predictors to get the model matrix A. The OLS solution solves the normal equations:

(Aᵀ A) b = Aᵀ y

In practice you never invert AᵀA yourself; numerical libraries use a QR or SVD factorisation, which is more stable. In NumPy that is np.linalg.lstsq.

Interpreting a coefficient

The key phrase is "holding the other predictors fixed". In a yield model

yield = 12 + 0.9*temp - 2.5*catalyst

0.9 means: among batches with the same catalyst concentration, each extra degree is associated with 0.9 percentage points more yield. It is not the slope you would get from plotting yield against temperature alone. If temperature and catalyst were correlated in the data (say, operators added more catalyst on hot days), the simple slope would mix the two effects together. Multiple regression separates them, as far as the data allow.

Two further points:

• The intercept is the predicted value when every predictor is 0. That is often far outside the data (0 °C in a reactor that runs at 60–90 °C), so it is estimated imprecisely and rarely meaningful on its own. Centring predictors (subtracting their mean) makes the intercept the prediction at an average batch.
• Units matter. A coefficient of 0.9 per °C is 1.62 per °F. To compare predictors on different scales, look at standardised coefficients or, better, at the change over a realistic range of each input.

Worked example in Python

You will do this by hand with NumPy in the activity. In day-to-day work, statsmodels gives the full table with standard errors, t-statistics, p-values and confidence intervals:

import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("pilot_plant.csv")        # columns: yield_pct, temp, cat
model = smf.ols("yield_pct ~ temp + cat", data=df).fit()
print(model.summary())
print(model.conf_int())                     # 95% intervals for each coefficient

A typical (abridged) summary for 40 runs:

              coef    std err      t     P>|t|    [0.025   0.975]
Intercept   15.19      2.37      6.40    0.000    10.4     20.0
temp         0.859     0.031    27.95    0.000     0.80     0.92
cat         -2.594     0.243   -10.65    0.000    -3.09    -2.10
R-squared: 0.959     Adj. R-squared: 0.957

How to read it:

1. Coefficient and interval. Each extra °C adds about 0.86 points of yield (95% CI 0.80 to 0.92), holding catalyst fixed. The interval is narrow because temperature varied a lot.
2. t and p. t = coef / std err. A tiny p-value says the coefficient is clearly not zero given the model. It does not say the effect is large or causal.
3. R² = 1 − SSE/SST is the share of the variation in y the model explains. Adding any predictor, even pure noise, never lowers R². Adjusted R² penalises extra terms and is fairer for comparing models of different size.

Categorical predictors and interactions

Regression handles categories through indicator (dummy) variables. In a formula, C(supplier) creates one 0/1 column per supplier except a baseline, and each coefficient is the difference from that baseline. An interaction such as temp:cat (or temp * cat, which adds both main effects too) lets the effect of temperature depend on the catalyst level. Interactions are exactly what factorial experiments are built to estimate, which you will see in module 2.

Common mistakes

• Reading coefficients causally from observational data. "Holding fixed" is a statement about the model, not about an intervention. Unmeasured variables can still drive both x and y (lesson 9 covers confounding).
• Ignoring collinearity. When two predictors are highly correlated, their individual coefficients become unstable and their standard errors balloon, even though predictions may be fine. Check correlations or variance inflation factors (VIFs; values above about 5–10 are a warning).
• Extrapolating. The fitted plane is only supported where you have data. Predicting yield at 120 °C from runs between 60 and 90 °C is a guess.
• Chasing R². A high R² does not mean the model is right; a low R² does not mean the effect is unimportant. Look at residuals (next lesson) and at whether the coefficients make scientific sense.
• Throwing in every available column. More predictors means more variance, more chances of a spurious "significant" result and harder interpretation. Choose predictors with the subject-matter expert.

Quick recap

• Multiple regression fits y = b0 + b1*x1 + ... + bp*xp by least squares.
• Each slope is the expected change in y per unit of that predictor, holding the others fixed.
• Use confidence intervals, not just p-values; use adjusted R² to compare model sizes.
• Watch for collinearity, extrapolation and causal over-reading.

Try it

Open the Do tab. You will build the model matrix, solve least squares with NumPy and compute R² for 40 simulated pilot-plant runs, then compare your estimates with the true values used to generate the data.