Training deep models

Why this matters

Getting a deep model to train is the easy part. Getting it to generalise, to work on data it has never seen, is the job. Every training run ends in one of three states: the model is too simple to fit even the training data (underfitting), it fits the training data but not new data (overfitting), or it is about right. Reading those states from loss curves, and knowing which lever to pull for each, is the most practical skill in applied deep learning.

The core idea: compare training and validation loss

Keep a validation set the model never trains on (you met this split in Machine Learning Foundations). After each epoch, record both losses and look at two things:

1. Is the training loss low enough? Compare it with a target: a simple baseline, human-level error, or the error your use case can tolerate.
2. Is there a gap? Validation loss much higher than training loss means the model memorised details that do not carry over.

train high,  val high     -> underfitting  (high bias)
train low,   val high     -> overfitting   (high variance)
train low,   val close    -> good fit

Worked example

You fit polynomials to 15 noisy points from sin(3x):

The degree-12 model has enough capacity to pass near every training point, including the noise, and swings wildly between them. The penalty keeps its weights small so the curve stays smooth. The noise in this data has a variance of 0.04, so no model can do much better than a validation MSE of 0.04. You will reproduce this table in the activity.

Fixing underfitting

• More capacity: more layers or units, a richer model family.
• Train longer, or fix the learning rate (too low a rate looks like underfitting).
• Better features or less regularisation.

Fixing overfitting

• More data, or data augmentation (flipped or cropped images, paraphrased text), which is often the most effective fix.
• L2 regularisation (weight decay): add lam * sum(w**2) to the loss. For a linear model this has a closed form, ridge regression:

import numpy as np

def ridge_fit(X, y, lam):
    # minimises ||Xw - y||^2 + lam * ||w||^2
    return np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ y)

Larger lam means smaller weights and a smoother model. In PyTorch you pass weight_decay=1e-4 to the optimiser; in Keras you add kernel_regularizer=keras.regularizers.l2(1e-4) to a layer.

• Dropout: during training, randomly zero a fraction of hidden units (for example 20–50%) so the network cannot rely on any single unit. It is switched off at prediction time.
• Early stopping: stop when the validation loss stops improving, and keep the best weights.
• Smaller model, as a last resort.

Early stopping in detail

Validation loss usually falls, flattens, then rises as the model starts to memorise. Early stopping watches it with a patience setting:

epoch:     0     1     2     3     4     5     6     7
val loss: 0.90  0.60  0.45  0.40  0.38  0.39  0.41  0.40
                                   best   +1    +2    +3 -> stop (patience 3)

Epoch 4 is the best, and three epochs pass without beating it, so training stops and the epoch-4 weights are restored. Keras has keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True); in PyTorch you write the few lines yourself or use a library such as PyTorch Lightning.

Other training signals

• Loss is nan or explodes: learning rate too high, or bad input values. Lower the rate, check for missing values, and try gradient clipping.
• Loss flat from the start: the learning rate is too low, a bug zeroes the gradients, or the labels are shuffled relative to the inputs. Try to overfit a tiny batch of 10 examples first: a healthy model should drive the training loss near zero. If it cannot, there is a bug.
• Validation loss noisy: the validation set is too small. Use more data or cross-validation.

Common mistakes

• Tuning on the test set. Every decision (epochs, lam, architecture) uses validation data. Touch the test set once at the end.
• Reading training accuracy only. It always improves with more capacity.
• Applying dropout at evaluation. Call model.eval() in PyTorch before validating; Keras handles this for you.
• Adding regularisation to an underfitting model. It makes the problem worse.
• Changing several things at once. Change one lever per run and record it (module 4 covers experiment tracking).

Quick recap

• Underfitting: training loss itself is too high. Add capacity, train longer, regularise less.
• Overfitting: big gap between training and validation loss. Add data, augmentation, weight decay, dropout or early stopping.
• Early stopping keeps the weights from the best validation epoch.
• Before a long run, check the model can overfit a tiny batch.

Try it

Open the Do tab. You will write a diagnosis function, an early-stopping rule and ridge regression in NumPy, then show that an L2 penalty rescues an overfitting degree-12 model.