Reproducible training

Why this matters

A model in production is a promise: "this artefact, trained on this data with this code, behaves like the one we evaluated." If you cannot rebuild it, you cannot debug it, audit it or safely improve it. A surprising number of production incidents start with a sentence like "it worked on my laptop" or "we are not sure which data the current model was trained on." Reproducible training turns a one-off notebook result into something a team can trust.

The core idea: pin three things

Every trained model is a function of three inputs. Pin all three and you can rebuild the model.

1. Code: the exact commit of your training script, feature code and configuration.
2. Data: the exact snapshot of the training data (not "the customers table", but "the customers table as of 2026-10-01, hash 9f2c...").
3. Environment: the Python version, every library version and, ideally, the operating system image.

Add a fourth, smaller item: randomness. Shuffling, weight initialisation and train/test splits all use random numbers, so fix the seeds.

Pinning code

Commit everything that affects training, including configuration such as hyperparameters. Keep configuration in a file (config.yaml or params.json) rather than in notebook cells you edited by hand. When you train, record the commit hash:

git rev-parse HEAD

If the working tree has uncommitted changes, the hash alone does not describe the code. Many teams make the training script refuse to run on a "dirty" tree, or at least log a warning.

Pinning data

Data is large and changes, so you usually do not commit it to Git directly. Common options:

• Content hashes. Compute a SHA-256 of the data file and store it with the model. If the hash changes, the data changed.
• Data version control (DVC). DVC stores small pointer files in Git and the real data in a cache or remote storage, so git checkout plus dvc checkout restores matching code and data.
• Immutable snapshots. Write each training set to a dated, read-only location (for example s3://bucket/train/2026-10-01/) and never overwrite it.

import hashlib

def file_sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

Pinning the environment

pip install pandas gives you whatever version is current today. Six months later it gives you a different one. Pin exact versions:

# requirements.txt
numpy==2.1.3
pandas==2.2.3
scikit-learn==1.5.2

pip freeze > requirements.txt captures what is installed; lock-file tools (pip-tools, Poetry, uv) do the same with dependency resolution. A container image goes further and pins the operating system and system libraries too:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "train.py"]

Pin the base image as well (python:3.12-slim moves over time; a digest such as python:3.12-slim@sha256:... does not).

Recording a run

Experiment trackers such as MLflow save the parameters, metrics, code version and artefacts of each run. MLflow runs locally with no account: pip install mlflow, log from your script, and browse runs with mlflow ui at http://127.0.0.1:5000.

import mlflow

with mlflow.start_run():
    mlflow.log_params({"seed": 42, "max_depth": 6, "data_sha256": data_hash})
    mlflow.log_param("git_commit", commit)
    mlflow.log_metric("val_auc", auc)
    mlflow.log_artifact("requirements.txt")

Worked example: same inputs, same model

You train a gradient-boosted model twice on Monday and get validation AUC 0.871 both times. On Thursday a colleague gets 0.864. Because each run logged the commit, data hash and package versions, the diff takes one minute: the commit and data match, but their environment had a newer scikit-learn whose default changed. Pinning the version restores 0.871. Without the records, that investigation could take days.

Exact bit-for-bit equality is not always possible (GPU kernels and multi-threading can add tiny non-determinism). The practical goal is: same inputs give the same model within a small, known tolerance, and you can explain any difference.

Common mistakes

• Training from a live table. The data changes under you. Snapshot first, then train on the snapshot.
• Unpinned dependencies. "Latest" is not a version.
• Seeds set in one place only. Python's random, NumPy and your framework each have their own generator.
• Hand-edited notebooks. Cells run out of order cannot be replayed. Move training into a script.
• Not saving the preprocessing. The scaler, vocabulary or encoder is part of the model; version it with the model.

Quick recap

• A model is a function of code, data, environment and seeds: pin all four.
• Use Git commits, data hashes or DVC, exact package versions and a container image.
• Log every run (parameters, metrics, commit, data hash) with a tracker such as MLflow.
• Aim for "same inputs, same model within tolerance" and be able to explain any drift.

Try it

Open the Do tab and make a small training project fully reproducible on your own machine, then prove it by training twice.