Training pipelines

Why this matters

Models go stale. Customer behaviour changes, new products appear, sensors are replaced. Sooner or later you must retrain, and if retraining means "someone re-runs a notebook by hand", it will happen late, inconsistently and with mistakes. A training pipeline turns training into a repeatable, automated process that anyone on the team (or a scheduler) can run, and that produces a model you can compare against the one in production before you ship it.

The core idea: a graph of small, checkable steps

A pipeline is a directed acyclic graph (DAG) of steps. Each step has defined inputs and outputs, and runs only when its inputs are ready.

ingest -> validate -> features -> train -> evaluate -> register
                                              \-> (fail) stop and alert

Small steps make failures easy to locate and let you re-run only what changed.

Properties of a good pipeline

• Parameterised. The date range, data path and hyperparameters come from configuration, not from edits to the code.
• Idempotent. Running a step twice with the same inputs gives the same output and does not duplicate data. Write to a new versioned location instead of appending.
• Versioned outputs. Every artefact (data snapshot, features, model, report) is stored with an ID that links back to the code commit and inputs. This is the reproducibility lesson, automated.
• Gated. The pipeline does not deploy automatically just because training finished. The evaluate step compares the candidate to a baseline and to the production model on the same held-out data, and stops if it is worse.
• Observable. Each step logs its duration, row counts and key metrics so you can see where time and quality go.

Worked example: a retraining pipeline as plain Python

You do not need a platform to start. A pipeline can be a script whose functions are the steps:

# pipeline.py
import json, pathlib

def run(config):
    out = pathlib.Path("runs") / config["run_id"]
    out.mkdir(parents=True, exist_ok=False)          # new folder per run: idempotent
    raw = ingest(config["start"], config["end"])     # returns a DataFrame
    check_schema(raw)                                # raises on bad data
    X, y = build_features(raw)                       # shared with serving
    model = train(X, y, seed=config["seed"], **config["params"])
    report = evaluate(model, config["holdout_path"])
    (out / "report.json").write_text(json.dumps(report, indent=2))
    if report["auc"] < config["min_auc"]:
        raise SystemExit("candidate below threshold; not registering")
    save_model(model, out / "model.joblib")

When this grows, orchestrators take over the scheduling, retries and caching: Airflow, Prefect and Dagster for general data pipelines; Kubeflow Pipelines, Vertex AI Pipelines, SageMaker Pipelines or TFX for ML on managed or Kubernetes platforms; and lightweight options such as DVC pipelines (dvc.yaml) or a Makefile that rebuild only changed stages. The design (steps, inputs, outputs, gates) stays the same whichever tool you pick.

When to retrain

Most teams start with a schedule and add drift or performance triggers once monitoring is in place.

Common mistakes

• One giant step. A three-hour script that fails at the end and must be re-run from scratch.
• Silent data problems. No validation step, so an empty or mis-joined table trains a broken model.
• Auto-deploying. Shipping every newly trained model without comparing it to the one already in production.
• Different feature code. Training builds features one way and serving another (more on this next lesson).
• Overwriting outputs. Re-runs replace yesterday's artefacts, so you cannot compare or roll back.

Quick recap

• A pipeline is a DAG of small steps with defined inputs, outputs and checks.
• Make it parameterised, idempotent, versioned, gated and observable.
• Start with a script or Makefile; adopt an orchestrator when scheduling and retries get hard.
• Retrain on a schedule first, then on drift or performance signals.

Try it

Open the Do tab and design (and optionally run locally) a retrainable pipeline for a small model on your own machine.