Training pipelines

Why this matters

Models go stale. Customer behaviour changes, new products appear, sensors are replaced. Sooner or later you must retrain, and if retraining means "someone re-runs a notebook by hand", it will happen late, inconsistently and with mistakes. A training pipelineA repeatable, automated process that turns data into a candidate model through a series of steps. turns training into a repeatable, automated process that anyone on the team (or a scheduler) can run, and that produces a model you can compare against the one in production before you ship it.

The core idea: a graph of small, checkable steps

A pipeline is a directed acyclic graphA set of steps connected in one direction with no loops, where each step runs once its inputs are ready. (DAG) of steps. Each step has defined inputs and outputs, and runs only when its inputs are ready.

ingest -> validate -> features -> train -> evaluate -> register
                                              \-> (fail) stop and alert

Small steps make failures easy to locate and let you re-run only what changed.

Properties of a good pipeline

• Parameterised. The date range, data path and hyperparameters come from configuration, not from edits to the code.
• IdempotentDescribes a step that gives the same output and no duplicate data when run twice with the same inputs.. Running a step twice with the same inputs gives the same output and does not duplicate data. Write to a new versioned location instead of appending.
• Versioned outputs. Every artefact (data snapshot, features, model, report) is stored with an ID that links back to the code commit and inputs. This is the reproducibility lesson, automated.
• GatedDescribes a pipeline that stops instead of registering a candidate that does not beat the baseline and current model.. The pipeline does not deploy automatically just because training finished. The evaluate step compares the candidate to a baseline and to the production model on the same held-out data, and stops if it is worse.
• Observable. Each step logs its duration, row counts and key metrics so you can see where time and quality go.

Worked example: a retraining pipeline as plain Python

You do not need a platform to start. A pipeline can be a script whose functions are the steps:

# pipeline.py
import json, pathlib

def run(config):
    out = pathlib.Path("runs") / config["run_id"]
    out.mkdir(parents=True, exist_ok=False)          # new folder per run: idempotent
    raw = ingest(config["start"], config["end"])     # returns a DataFrame
    check_schema(raw)                                # raises on bad data
    X, y = build_features(raw)                       # shared with serving
    model = train(X, y, seed=config["seed"], **config["params"])
    report = evaluate(model, config["holdout_path"])
    (out / "report.json").write_text(json.dumps(report, indent=2))
    if report["auc"] < config["min_auc"]:
        raise SystemExit("candidate below threshold; not registering")
    save_model(model, out / "model.joblib")

When this grows, orchestratorsTools such as Airflow, Prefect or Dagster that schedule pipeline steps and handle retries and caching. take over the scheduling, retries and caching: Airflow, Prefect and Dagster for general data pipelines; Kubeflow Pipelines, Vertex AI Pipelines, SageMaker Pipelines or TFX for ML on managed or Kubernetes platforms; and lightweight options such as DVC pipelines (dvc.yaml) or a Makefile that rebuild only changed stages. The design (steps, inputs, outputs, gates) stays the same whichever tool you pick.

When to retrain

Most teams start with a schedule and add drift or performance triggers once monitoring is in place.

Common mistakes

• One giant step. A three-hour script that fails at the end and must be re-run from scratch.
• Silent data problems. No validation step, so an empty or mis-joined table trains a broken model.
• Auto-deploying. Shipping every newly trained model without comparing it to the one already in production.
• Different feature code. Training builds features one way and serving another (more on this next lesson).
• Overwriting outputs. Re-runs replace yesterday's artefacts, so you cannot compare or roll back.

Quick recap

• A pipeline is a DAG of small steps with defined inputs, outputs and checks.
• Make it parameterised, idempotent, versioned, gated and observable.
• Start with a script or Makefile; adopt an orchestrator when scheduling and retries get hard.
• Retrain on a schedule first, then on drift or performance signals.

Try it

Open the Do tab and design (and optionally run locally) a retrainable pipeline for a small model on your own machine.