Train, validation, and test

Why this matters

A model that has memorised its training data can look perfect and still fail on the next customer. The only honest way to know how a model will do on new data is to test it on data it has never seen, not during training, not during tuning, not while you chose features. That is what splitting is for, and getting it wrong is the most common reason ML results do not hold up.

Three sets, three jobs

• Training set (often 60–80%): the model's parameters are fitted on it.
• Validation set (10–20%): you compare models and choose settings (hyperparameters) with it.
• Test set (10–20%): you look at it once, at the end, to report expected performance.

Why three and not two? Every time you check a score and change something, a little information about that set flows into your choices. After you have tried 50 settings against the validation set, its score is optimistic. The test set, untouched until the end, gives the unbiased estimate. In Lesson 12 you will replace the fixed validation set with cross-validation.

Shuffle before you split (usually)

Data files are often sorted: by date, by region, by label. If you take the first 80% of a file sorted by label as training data, the test set may contain only one class. So you shuffle the row order with a fixed random seed (so the split is reproducible) and then cut.

import numpy as np

rng = np.random.default_rng(42)
idx = rng.permutation(10)      # a shuffled 0..9, the same every run with seed 42
test_idx, train_idx = idx[:2], idx[2:]

Index arrays are the cleanest way to split: apply the same indices to X and y so rows and labels stay aligned.

Two important exceptions:

• Time series: never shuffle. Train on the past, test on the future (for example train on January–September, test on October). Shuffling would let the model peek at the future.
• Groups: if one patient has 20 rows, all 20 must land in the same set. Otherwise the model recognises the patient, not the pattern. scikit-learn has GroupShuffleSplit for this.

For classification with rare classes, use a stratified split so each set keeps the same class proportions (train_test_split(..., stratify=y)).

Data leakage

Leakage is any way information from the validation or test data, or from the future, sneaks into training. It inflates scores and the model then disappoints in production. Typical leaks:

1. Preprocessing fitted on all the data. Computing the mean and standard deviation for scaling over the full dataset before splitting lets test statistics shape training. Fit preprocessing on the training set only, then apply those same numbers to validation and test.
2. Target leakage. A feature that is a consequence of the label: "account closed date" when predicting churn, or "treatment given" when predicting diagnosis.
3. Duplicate or near-duplicate rows across sets.
4. Feature selection on the full dataset, for example keeping the 20 columns most correlated with the label before splitting.

Worked example: scaling without leakage

X_train, X_test = X[train_idx], X[test_idx]

mean = X_train.mean(axis=0)      # statistics from TRAIN only
std = X_train.std(axis=0)

X_train_s = (X_train - mean) / std
X_test_s = (X_test - mean) / std   # same numbers, applied to test

The scaled training columns have mean 0 and standard deviation 1. The scaled test columns will be close to that but not exactly, and that is correct: it is how new data will look in production.

The scikit-learn way

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

scaler = StandardScaler().fit(X_train)     # fit on train only
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)

A Pipeline (Lesson 10) bundles the scaler and the model so the "fit on train only" rule is applied automatically, even inside cross-validation.

Common mistakes

• Tuning on the test set. If you look at test scores while choosing settings, the test set has become a second validation set.
• Splitting after preprocessing, which leaks statistics.
• Random splits of time-ordered data, which lets the model use the future.
• No seed, so nobody (including you next week) can reproduce the numbers.
• Splitting rows when the real unit is a person, device or store.

Quick recap

• Train fits parameters, validation chooses settings, test is touched once.
• Shuffle with a fixed seed, except for time series (split by time) and grouped data (split by group).
• Fit every preprocessing step on the training set only.
• If a score looks too good to be true, look for leakage first.

Try it

On the Do tab, write a reproducible shuffled split and a scaler that is fitted on the training rows only.