Reproducible training

Why this matters

A model in production is a promise: "this artefact, trained on this data with this code, behaves like the one we evaluated." If you cannot rebuild it, you cannot debug it, audit it or safely improve it. A surprising number of production incidents start with a sentence like "it worked on my laptop" or "we are not sure which data the current model was trained on." Reproducible training turns a one-off notebook result into something a team can trust.

The core idea: pin three things

Every trained model is a function of three inputs. Pin all three and you can rebuild the model.

1. Code: the exact commit of your training script, feature code and configuration.
2. Data: the exact snapshot of the training data (not "the customers table", but "the customers table as of 2026-10-01, hash 9f2c...").
3. Environment: the Python version, every library version and, ideally, the operating system image.

Add a fourth, smaller item: randomness. Shuffling, weight initialisation and train/test splits all use random numbers, so fix the seeds.

Pinning code

Commit everything that affects training, including configuration such as hyperparameters. Keep configuration in a file (config.yaml or params.json) rather than in notebook cells you edited by hand. When you train, record the commit hashThe identifier Git gives an exact version of the code, recorded with each training run.:

git rev-parse HEAD

If the working tree has uncommitted changes, the hash alone does not describe the code. Many teams make the training script refuse to run on a "dirty" tree, or at least log a warning.

Pinning data

Data is large and changes, so you usually do not commit it to Git directly. Common options:

• Content hashesFingerprints such as SHA-256 computed from a data file, which change whenever the data changes.. Compute a SHA-256 of the data file and store it with the model. If the hash changes, the data changed.
• Data version control (DVC)A tool that keeps small pointer files in Git and the real data in a cache or remote storage, so code and data can be restored together.. DVC stores small pointer files in Git and the real data in a cache or remote storage, so git checkout plus dvc checkout restores matching code and data.
• Immutable snapshots. Write each training set to a dated, read-only location (for example s3://bucket/train/2026-10-01/) and never overwrite it.

import hashlib

def file_sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

Pinning the environment

pip install pandas gives you whatever version is current today. Six months later it gives you a different one. Pin exact versions:

# requirements.txt
numpy==2.1.3
pandas==2.2.3
scikit-learn==1.5.2

pip freeze > requirements.txt captures what is installed; lock-file tools (pip-tools, Poetry, uv) do the same with dependency resolution. A container imageA packaged environment that pins the operating system, system libraries and Python packages a program runs with. goes further and pins the operating system and system libraries too:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "train.py"]

Pin the base image as well (python:3.12-slim moves over time; a digest such as python:3.12-slim@sha256:... does not).

Recording a run

Experiment trackers such as MLflowAn experiment tracker that saves the parameters, metrics, code version and artefacts of each run. save the parameters, metrics, code version and artefacts of each run. MLflow runs locally with no account: pip install mlflow, log from your script, and browse runs with mlflow ui at http://127.0.0.1:5000.

import mlflow

with mlflow.start_run():
    mlflow.log_params({"seed": 42, "max_depth": 6, "data_sha256": data_hash})
    mlflow.log_param("git_commit", commit)
    mlflow.log_metric("val_auc", auc)
    mlflow.log_artifact("requirements.txt")

Worked example: same inputs, same model

You train a gradient-boosted model twice on Monday and get validation AUC 0.871 both times. On Thursday a colleague gets 0.864. Because each run logged the commit, data hash and package versions, the diff takes one minute: the commit and data match, but their environment had a newer scikit-learn whose default changed. Pinning the version restores 0.871. Without the records, that investigation could take days.

Exact bit-for-bit equality is not always possible (GPU kernels and multi-threading can add tiny non-determinism). The practical goal is: same inputs give the same model within a small, known tolerance, and you can explain any difference.

Common mistakes

• Training from a live table. The data changes under you. Snapshot first, then train on the snapshot.
• Unpinned dependencies. "Latest" is not a version.
• Seeds set in one place only. Python's random, NumPy and your framework each have their own generator.
• Hand-edited notebooks. Cells run out of order cannot be replayed. Move training into a script.
• Not saving the preprocessing. The scaler, vocabulary or encoder is part of the model; version it with the model.

Quick recap

• A model is a function of code, data, environment and seeds: pin all four.
• Use Git commits, data hashes or DVC, exact package versions and a container image.
• Log every run (parameters, metrics, commit, data hash) with a tracker such as MLflow.
• Aim for "same inputs, same model within tolerance" and be able to explain any drift.

Try it

Open the Do tab and make a small training project fully reproducible on your own machine, then prove it by training twice.