Linear and logistic regression

Why this matters

Linear regression and logistic regression are the workhorses of applied machine learning. They train in milliseconds, are hard to overfit badly, and, most importantly, you can explain them: each feature gets one coefficient that says how it moves the prediction. Many production systems in banking, healthcare and marketing still use them for exactly that reason, and they are the baseline every fancier model has to beat.

Linear regression

For a numeric label, linear regression predicts a weighted sum of the features:

y_hat = b + w1*x1 + w2*x2 + ... + wp*xp        (in matrix form: y_hat = X @ w + b)

Training chooses w and b to minimise the mean squared error, exactly as in the gradient descent lesson. For linear regression there is also an exact solution (least squares), which NumPy computes directly:

import numpy as np

A = np.column_stack([np.ones(len(X)), X])     # a column of 1s for the intercept
coef, *_ = np.linalg.lstsq(A, y, rcond=None)
b, w = coef[0], coef[1:]

Interpreting coefficients

Suppose a model of apartment rent (in dollars per month) gives

rent_hat = 400 + 2.1 * size_sqft + 150 * bedrooms - 12 * minutes_to_station

Read each coefficient as: holding the other features fixed, one more unit of this feature changes the prediction by this amount. One extra minute from the station is associated with $12 less rent, for apartments of the same size and bedroom count.

Three cautions:

• Coefficients depend on units. Measure size in square metres and its coefficient grows about 10.8 times. To compare importance across features, standardise them first.
• Correlated features share credit unpredictably. If size and bedrooms move together, their individual coefficients become unstable even when predictions are fine.
• Coefficients describe association, not cause.

The usual quality measures are RMSE (typical error in the label's units) and R-squared (the share of variance explained).

Logistic regression

For a yes/no label, we want a probability between 0 and 1. Logistic regression takes the same weighted sum and squashes it with the sigmoidThe function 1 / (1 + e^(-z)) that squashes any real number into a probability between 0 and 1. function:

z = X @ w + b                 (the log-odds, any real number)
p = sigmoid(z) = 1 / (1 + e^(-z))   (a probability)

sigmoid(0) = 0.5, large positive z gives probabilities near 1, large negative z near 0.

The loss: log loss

Squared error works poorly for probabilities. Logistic regression minimises log lossThe loss for probability predictions (binary cross-entropy), which punishes confident wrong answers heavily. (binary cross-entropy):

log_loss = -mean( y*log(p) + (1-y)*log(1-p) )

It punishes confident wrong answers heavily: predicting p = 0.01 for a true positive costs -log(0.01) = 4.6, while p = 0.6 costs only 0.51. There is no closed-form solution, so we use gradient descent. Conveniently, the gradient has the same shape as for linear regression:

err = p - y
dw = X.T @ err / n        db = mean(err)

Interpreting logistic coefficients

A coefficient is the change in log-oddsThe weighted sum z in logistic regression, which the sigmoid turns into a probability. per unit of the feature. Exponentiate it to get an odds ratioThe exponentiated logistic coefficient: how much one extra unit of a feature multiplies the odds.. In a churn model, a coefficient of 0.7 on "support tickets last month" means each extra ticket multiplies the odds of churning by e^0.7 ≈ 2.0, holding other features fixed.

From probability to decision

The model outputs probabilities; a thresholdThe probability cut-off that turns a model's output into a yes/no decision; 0.5 is only the default. turns them into decisions. 0.5 is only the default. If missing a churner is expensive and a retention email is cheap, use a lower threshold such as 0.3. The next lesson shows how to choose it with metrics.

Worked example in scikit-learn

from sklearn.linear_model import LinearRegression, LogisticRegression

lin = LinearRegression().fit(X_train, y_rent)
print(lin.intercept_, lin.coef_)

clf = LogisticRegression().fit(X_train, y_churn)
proba = clf.predict_proba(X_test)[:, 1]      # P(class 1)
pred = (proba >= 0.3).astype(int)            # your own threshold

Note that scikit-learn's LogisticRegression applies L2 regularisation by default (C=1.0), which shrinks coefficients toward zero. That is usually good, but it means coefficients differ slightly from an unregularised fit, and it makes feature scaling matter.

Common mistakes

• Using linear regression for a 0/1 label. It can predict probabilities below 0 or above 1. Use logistic regression.
• Comparing raw coefficients across features with different units.
• Forgetting the intercept, which forces the line through the origin.
• Overflow in the sigmoid. np.exp(-z) overflows for very negative z; clip z (for example to ±500) or use a library implementation.
• Reading coefficients as causes. They tell you what the model uses, not what would happen if you intervened.

Quick recap

• Linear regression: weighted sum, minimise MSE, exact least-squares solution, coefficients in label units per feature unit.
• Logistic regression: weighted sum through a sigmoid, minimise log loss with gradient descent, coefficients in log-odds (exponentiate for odds ratios).
• The threshold is a business decision, not a fixed 0.5.

Try it

On the Do tab, write the sigmoid and the log-loss gradient, train a logistic regression from scratch, and interpret its coefficients.