Cross-validation and tuning

Why this matters

Most models have hyperparameters: settings you choose before training rather than parameters the model learns. The regularisation strength of a ridge regression, the depth of a tree, k in k-NN, the number of trees in a forest, the learning rate of gradient descent. Picking them well can matter as much as picking the model. Picking them honestly, without peeking at the test set, is what keeps your reported performance true.

The problem with one validation split

In Lesson 2 you held out one validation set. With a small dataset, that single split is noisy: a different random split can rank the settings differently, and you may tune to the quirks of those particular rows. k-fold cross-validation uses the training data more efficiently and gives a more stable estimate.

k-fold cross-validation

shuffle the training rows and cut them into k folds (often k = 5 or 10)
for each fold i:
    train on the other k-1 folds
    score on fold i
CV score = average of the k scores

Every training row is used for validation exactly once and for training k - 1 times. The spread of the k scores also tells you how uncertain the estimate is.

import numpy as np

idx = np.random.default_rng(0).permutation(n)
folds = np.array_split(idx, 5)           # 5 nearly equal folds
for i in range(5):
    val_idx = folds[i]
    train_idx = np.concatenate(folds[:i] + folds[i + 1:])

The same rules as before apply: stratify folds for imbalanced classes, use group folds when rows belong to the same person, and use forward-chaining (expanding-window) splits for time series. Any preprocessing must be fitted inside each fold, which is why scikit-learn pipelines matter.

Grid search

To tune, try a list of candidate values, compute the CV score of each, and pick the best. These are the numbers from this lesson's activity (ridge regression on 40 noisy points):

alpha      CV MSE
1e-08      16.57   <- too flexible: fits the noise, wild between points
1e-04       0.14
1e-02       0.06   <- best
1           0.11
100         0.37   <- too rigid: underfits

Then refit on the whole training set with the chosen value and evaluate once on the test set.

Search on a logarithmic scale for strengths and rates (1e-4, 1e-3, ... 10) because their effects are multiplicative. For many hyperparameters at once, random search often finds good settings faster than a full grid, and Bayesian optimisation tools (such as Optuna) go further.

Ridge regression and the bias-variance trade-off

The activity tunes ridge regression, which is linear regression plus a penalty alpha * sum(w^2) on large coefficients. The exact solution is

w = (Xc^T Xc + alpha * I)^(-1) Xc^T yc        (Xc, yc centred, so the intercept is not penalised)

With 12 polynomial features of x, a tiny alpha lets the curve wiggle through every noisy point: low bias, high variance, and poor predictions on new data (overfitting). A huge alpha shrinks the curve toward a flat line: low variance, high bias (underfitting). Cross-validation finds the sweet spot between them. Every hyperparameter that controls flexibility (tree depth, k in k-NN, number of boosting rounds) is a version of this dial.

Nested cross-validation

If you report the best CV score from a big search as your final performance, it is optimistic: you picked the maximum of many noisy numbers. Either evaluate once on an untouched test set (as here), or, when data is scarce, use nested cross-validation: an inner loop chooses hyperparameters, and an outer loop scores the whole tuning procedure.

In scikit-learn

from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

pipe = make_pipeline(PolynomialFeatures(12, include_bias=False), StandardScaler(), Ridge())
search = GridSearchCV(
    pipe, {"ridge__alpha": [1e-8, 1e-6, 1e-4, 1e-2, 1, 100]},
    cv=KFold(5, shuffle=True, random_state=0), scoring="neg_mean_squared_error")
search.fit(X_train, y_train)
print(search.best_params_, -search.best_score_)
print(search.score(X_test, y_test))           # once, at the end

GridSearchCV refits the best pipeline on all the training data automatically. RandomizedSearchCV does random search, and cross_val_score gives the k fold scores for one fixed setting.

Common mistakes

• Tuning on the test set, directly or by trying "just one more" setting after seeing test results.
• Preprocessing outside the folds (for example scaling all training rows before cross-validation), which leaks validation statistics.
• Shuffled folds on time series.
• Searching a linear grid (0.1, 0.2, 0.3...) for a parameter that varies over orders of magnitude.
• Ignoring the spread of fold scores: if two settings differ by less than the fold-to-fold noise, prefer the simpler model.

Quick recap

• Hyperparameters are chosen, not learned; choose them with cross-validation on training data.
• k-fold CV trains k times, validating on each fold once, and averages the scores.
• Grid or random search over log-scaled candidates, refit on all training data, test once.
• Regularisation strength trades variance against bias; CV finds the balance.

Try it

On the Do tab, write k-fold splitting and cross-validated MSE, then let a grid search choose the ridge penalty.