Gradient descent

Why this matters

Almost every model in machine learning is trained the same way: define a loss that measures how wrong the predictions are, then adjust the parameters step by step to make the loss smaller. The step-by-step method is gradient descent. Linear and logistic regression, neural networks and gradient-boosted trees all rely on it or on a close relative. If you understand it for a straight line, you understand the core of training.

The setup: fitting a line

Model: y_hat = w * x + b, with two parameters, the slope w and the intercept b.

Loss: mean squared error over the n training points.

MSE(w, b) = (1/n) * sum( (w*x_i + b - y_i)^2 )

The MSE is a bowl-shaped surface over the (w, b) plane. We want the bottom of the bowl.

The gradient

The gradient is the vector of partial derivatives. It points uphill, in the direction where the loss grows fastest. With the error e_i = w*x_i + b - y_i, the chain rule gives

dMSE/dw = (2/n) * sum( e_i * x_i )
dMSE/db = (2/n) * sum( e_i )

If the predictions are too high on average, dMSE/db is positive, so we should decrease b. Gradient descent does exactly that: step against the gradient.

The algorithm

start with w = 0, b = 0
repeat for a number of steps:
    compute the gradient (dw, db)
    w = w - learning_rate * dw
    b = b - learning_rate * db

In NumPy, the whole dataset is processed at once (vectorised), so each step is a few lines:

import numpy as np

def gradients(w, b, x, y):
    err = w * x + b - y
    return 2 * np.mean(err * x), 2 * np.mean(err)

w, b, lr = 0.0, 0.0, 0.01
for step in range(5000):
    dw, db = gradients(w, b, x, y)
    w -= lr * dw
    b -= lr * db

Worked example: one step by hand

Data: x = [1, 2, 3], y = [2, 4, 6] (the true line is y = 2x). Start at w = 0, b = 0, learning rate 0.1.

errors e = [0-2, 0-4, 0-6] = [-2, -4, -6]
dw = (2/3) * (-2*1 + -4*2 + -6*3) = (2/3) * (-28) = -18.67
db = (2/3) * (-2 - 4 - 6)        = -8
w = 0 - 0.1 * (-18.67) = 1.867
b = 0 - 0.1 * (-8)     = 0.8

One step moved the slope from 0 most of the way to 2. Further steps shrink the remaining error.

The learning rate

The learning rate sets the step size and is the setting that matters most:

• Too small: progress is very slow; you need many steps.
• Too large: each step overshoots the bottom of the bowl, the loss grows, and the parameters blow up to inf or nan.
• About right: the loss falls quickly and then levels off.

Always record the loss at every step (the loss curve). A healthy curve goes down and flattens. A curve that rises or zigzags means the learning rate is too big.

Features on very different scales make the bowl long and narrow, which forces a small learning rate. Standardising features (Lesson 2) makes gradient descent converge much faster.

Checking your answer

For linear regression there is an exact solution (least squares), so you can check gradient descent against it:

w_exact, b_exact = np.polyfit(x, y, 1)    # degree-1 least-squares fit

After enough steps with a sensible learning rate, gradient descent agrees to several decimal places. In scikit-learn, LinearRegression uses the exact solution, while SGDRegressor uses stochastic gradient descent: it updates on small random batches of rows instead of the whole dataset, which is how models scale to millions of rows.

Variants you will meet

• Batch gradient descent: the full dataset per step (what you write today).
• Stochastic / mini-batch: a random subset per step; noisier but much cheaper per step.
• Momentum and Adam: keep a running average of past gradients to move faster along shallow directions. Adam is the default optimiser for neural networks.

Common mistakes

• Adding the gradient instead of subtracting it, which climbs the bowl.
• Forgetting the 2/n or using a sum instead of a mean, which silently changes the effective learning rate as the dataset grows.
• Updating w before computing the gradient for b, so the two gradients come from different points. Compute both, then update both.
• Not plotting or printing the loss, so divergence goes unnoticed.

Quick recap

• Training = minimising a loss. Gradient descent repeatedly steps against the gradient.
• For a line with MSE: dw = 2*mean(e*x), db = 2*mean(e).
• The learning rate controls stability and speed; watch the loss curve.
• Scaled features and mini-batches make gradient descent practical at scale.

Try it

On the Do tab, implement the gradient and the training loop, then check your line against the exact least-squares fit.