Take models from notebook to production: reproducible training, serving APIs, retrainable pipelines, CI/CD for ML, drift detection, monitoring and governance.

MLOps: Production Machine Learning

Write one sentence describing the model and the decision it supports, and how often its data changes.

Draw the pipeline as a DAG on paper or in a text file: at least ingest, validate, features, train, evaluate and register steps.

For each step, write its inputs, outputs (with a versioned path such as `runs/<run_id>/...`) and the check that must pass before the next step runs.

Define the evaluation gate: the metric, the baseline it must beat, and the maximum allowed drop compared with the current production model.

Choose a retraining trigger (schedule, performance, drift or data volume) and write down why it fits this model.

Implement the steps as functions in a `pipeline.py` script (or as stages in a `Makefile` or `dvc.yaml`), with all settings read from a config file.

Run it twice with the same config and confirm the second run either reuses or exactly reproduces the first run's outputs without overwriting them.

Break it on purpose (for example, remove a column from the input data) and confirm the validate step stops the run with a clear message.

Clean up: delete old run folders you do not need and stop any local services you started.

## Why this matters

Models go stale. Customer behaviour changes, new products appear, sensors are replaced. Sooner or later you must retrain, and if retraining means "someone re-runs a notebook by hand", it will happen late, inconsistently and with mistakes. A **training pipeline** turns training into a repeatable, automated process that anyone on the team (or a scheduler) can run, and that produces a model you can compare against the one in production before you ship it.

## The core idea: a graph of small, checkable steps

A pipeline is a directed acyclic graph (DAG) of steps. Each step has defined inputs and outputs, and runs only when its inputs are ready.

```text
ingest -> validate -> features -> train -> evaluate -> register
                                              \-> (fail) stop and alert
```

| step | input | output | what it checks |
|---|---|---|---|
| ingest | source tables, date range | raw snapshot + hash | row counts are plausible |
| validate | raw snapshot | validated data | schema, nulls, value ranges |
| features | validated data | feature table | same code as serving |
| train | features, config, seed | model artefact | training finished, loss sane |
| evaluate | model, held-out data | metrics report | beats a baseline and the current model |
| register | model + report | registry entry ("candidate") | a human or rule approves |

Small steps make failures easy to locate and let you re-run only what changed.

## Properties of a good pipeline

- **Parameterised.** The date range, data path and hyperparameters come from configuration, not from edits to the code.
- **Idempotent.** Running a step twice with the same inputs gives the same output and does not duplicate data. Write to a new versioned location instead of appending.
- **Versioned outputs.** Every artefact (data snapshot, features, model, report) is stored with an ID that links back to the code commit and inputs. This is the reproducibility lesson, automated.
- **Gated.** The pipeline does not deploy automatically just because training finished. The evaluate step compares the candidate to a baseline and to the production model on the same held-out data, and stops if it is worse.
- **Observable.** Each step logs its duration, row counts and key metrics so you can see where time and quality go.

## Worked example: a retraining pipeline as plain Python

You do not need a platform to start. A pipeline can be a script whose functions are the steps:

```python
# pipeline.py
import json, pathlib

def run(config):
    out = pathlib.Path("runs") / config["run_id"]
    out.mkdir(parents=True, exist_ok=False)          # new folder per run: idempotent
    raw = ingest(config["start"], config["end"])     # returns a DataFrame
    check_schema(raw)                                # raises on bad data
    X, y = build_features(raw)                       # shared with serving
    model = train(X, y, seed=config["seed"], **config["params"])
    report = evaluate(model, config["holdout_path"])
    (out / "report.json").write_text(json.dumps(report, indent=2))
    if report["auc"] < config["min_auc"]:
        raise SystemExit("candidate below threshold; not registering")
    save_model(model, out / "model.joblib")
```

When this grows, orchestrators take over the scheduling, retries and caching: **Airflow**, **Prefect** and **Dagster** for general data pipelines; **Kubeflow Pipelines**, **Vertex AI Pipelines**, **SageMaker Pipelines** or **TFX** for ML on managed or Kubernetes platforms; and lightweight options such as **DVC pipelines** (`dvc.yaml`) or a `Makefile` that rebuild only changed stages. The design (steps, inputs, outputs, gates) stays the same whichever tool you pick.

## When to retrain

| trigger | example | trade-off |
|---|---|---|
| schedule | every Sunday night | simple; may retrain when nothing changed |
| performance | live accuracy falls below a threshold | needs labels, which may arrive late |
| drift | input distribution shifts (module 3) | catches change early; can raise false alarms |
| new data volume | 100,000 new labelled rows | good for fast-growing products |

Most teams start with a schedule and add drift or performance triggers once monitoring is in place.

## Common mistakes

- **One giant step.** A three-hour script that fails at the end and must be re-run from scratch.
- **Silent data problems.** No validation step, so an empty or mis-joined table trains a broken model.
- **Auto-deploying.** Shipping every newly trained model without comparing it to the one already in production.
- **Different feature code.** Training builds features one way and serving another (more on this next lesson).
- **Overwriting outputs.** Re-runs replace yesterday's artefacts, so you cannot compare or roll back.

## Quick recap

- A pipeline is a DAG of small steps with defined inputs, outputs and checks.
- Make it parameterised, idempotent, versioned, gated and observable.
- Start with a script or `Makefile`; adopt an orchestrator when scheduling and retries get hard.
- Retrain on a schedule first, then on drift or performance signals.

## Try it

Open the **Do** tab and design (and optionally run locally) a retrainable pipeline for a small model on your own machine.


Design a retrainable training pipeline of small, versioned, gated steps from ingestion to model registration.

Training pipelines

Intro to Kubeflow Pipelines

How to build an ML pipeline with TFX

## Why this matters

A model can score 0.90 AUC offline and behave far worse in production, with no bug in the model at all. Very often the cause is that the features it sees in production are not quite the features it was trained on. This gap is called **training/serving skew**, and it is one of the most common and most expensive failures in applied ML because nothing crashes: the model just quietly makes worse decisions.

## The core idea: training/serving skew

Skew means the feature values the model receives at prediction time differ from what the same row would have produced in training. It has three usual causes.

1. **Two implementations.** Data scientists compute `avg_order_value_30d` in SQL for training; engineers re-implement it in Java for the API. One rounds, one includes refunds, and the values drift apart.
2. **Different data freshness.** Training uses a complete, cleaned warehouse table; serving uses a live table where today's orders are still arriving.
3. **Point-in-time leakage.** Training rows accidentally use information from *after* the moment the prediction would have been made. Offline accuracy looks great because the model is peeking at the future, which it can never do in production.

### Worked example: leakage by a careless join

You predict on 1 March whether a customer will churn in March. Your training query joins each customer to their *current* `total_support_tickets`, computed today in October. Customers who churned had a burst of tickets in March and April, so the feature is strongly predictive offline. In production on 1 March, those future tickets do not exist yet. The fix is a **point-in-time correct** join: for each training row, use only feature values with a timestamp at or before the prediction time.

```sql
-- point-in-time join: latest feature value at or before the label's cut-off
SELECT l.customer_id, l.cutoff_date, f.tickets_30d
FROM labels l
JOIN features f
  ON f.customer_id = l.customer_id
 AND f.as_of = (SELECT MAX(as_of) FROM features
                WHERE customer_id = l.customer_id AND as_of <= l.cutoff_date);
```

## What a feature store does

A **feature store** is a system for defining features once and serving the same values to training and to production. The main parts:

- **Feature definitions**: the transformation code, written once and registered with a name, owner and version (for example `customer.tickets_30d`).
- **Offline store**: the full history of feature values with timestamps, usually in a warehouse or files, used to build training sets with point-in-time joins.
- **Online store**: the latest value per entity in a fast key-value database, so an API can fetch features in a few milliseconds.
- **Materialisation**: jobs that compute features and write them to both stores from the same definition.
- **Registry and discovery**: a catalogue so another team can reuse `tickets_30d` instead of rebuilding it slightly differently.

Examples include the open-source **Feast** and **Hopsworks**, and managed stores in Databricks, Vertex AI and SageMaker.

```text
           feature definition (one piece of code)
                 /                         \
     offline store (history)        online store (latest)
          |                                |
  point-in-time training set        low-latency lookup in the API
```

## Do you need one?

Not always. A feature store earns its cost when several models share features, when online serving needs fresh precomputed aggregates, or when point-in-time correctness is hard to get right by hand. A single batch model can avoid skew more cheaply by **sharing one feature function** between the training pipeline and the scoring job, and by logging the features actually used at prediction time so you can compare them with training data.

## Catching skew

- **Log serving features.** Store the exact feature vector each prediction used.
- **Compare distributions.** Regularly compare logged serving features with the training set (the drift tests in module 3 work here too).
- **Replay test.** Take a sample of production requests, recompute their features with the training code, and check that the values match within a tolerance.

## Common mistakes

- **Rewriting feature logic for production** instead of reusing the training code.
- **Joining on "current" values** when building historical training data.
- **Ignoring defaults.** Training drops rows with missing values; serving fills them with 0. The model never saw zeros there.
- **Buying a platform first.** Adopting a feature store before you have a feature-sharing problem.

## Quick recap

- Training/serving skew: production feature values differ from training ones, so performance drops silently.
- Causes: two implementations, different freshness, and point-in-time leakage.
- Feature stores define features once and serve them to an offline store (history) and an online store (latest).
- Small teams can share one feature function and log serving features instead.

## Try it

Open the **Do** tab and take the quiz to check you can spot skew and leakage in realistic setups.


Explain training/serving skew and point-in-time leakage, and when a feature store helps prevent them.

Feature stores

What is a Feature Store for Machine Learning?

MFML 082 - The training-serving skew

Install pytest and create a `tests/` folder. Add one unit test for a feature function, including an edge case (zero, missing value or unseen category).

Save a small fixed sample of the data (a few hundred rows you are allowed to share) in `tests/data/` and write a data test that checks the column names, value ranges and label values.

Write a model quality test: train on the sample with a fixed seed and assert the metric is above a floor that beats a simple baseline (for example the majority class) by a margin.

Write one directional or invariance test (for example, more support tickets should not lower churn risk).

Run `pytest -q` locally until everything passes, then break the data or model on purpose and confirm a test fails.

Create `.github/workflows/ml-ci.yml` that checks out the code, sets up Python with pip caching, installs pinned requirements and runs pytest on push and pull request.

Push to a GitHub repository, open a pull request, and confirm the workflow runs and its status appears on the pull request.

Clean up: if you used a private repository, check the Actions usage on your billing page; delete the repository or disable the workflow if you no longer need it.

## Why this matters

In ordinary software, continuous integration (CI) runs tests on every change so bugs are caught before they ship. ML systems need the same safety net, plus more: a change can break the code, the data or the model's quality, and only the first of those raises an exception. CI/CD for ML adds automated checks for data and model behaviour so that a "small refactor" cannot quietly cost you five points of accuracy in production.

## The core idea: test code, data and model

Think of three layers of tests, from fast to slow.

| layer | examples | when it runs |
|---|---|---|
| **code** | unit tests for feature functions, the API's input validation, config parsing | every push, seconds |
| **data** | schema, null rates, value ranges, row counts on a fixed sample | every push (sample) and every pipeline run (full) |
| **model** | minimum metric on a fixed holdout, comparison with the current model, behavioural tests | on changes to training code or data; minutes |

### Code tests

Feature code is ordinary code. Test it like any other function, especially the edge cases that silently break models: empty inputs, missing values, time zones and unseen categories.

```python
def test_tenure_bucket_handles_zero():
    assert tenure_bucket(0) == "0-3m"
```

### Data tests

Assert the assumptions your model depends on. Tools such as Great Expectations or pandera make this declarative, but plain `assert` works.

```python
def test_training_sample_schema():
    df = load_sample("tests/data/sample.csv")
    assert list(df.columns) == ["tenure_months", "monthly_spend", "support_tickets", "churned"]
    assert df["monthly_spend"].between(0, 100_000).all()
    assert df["churned"].isin([0, 1]).all()
```

### Model tests

- **Quality gate:** train on a small fixed dataset with a fixed seed and assert the metric is above a floor (for example AUC at least 0.80). Keep it small so CI stays fast.
- **Regression against production:** in the full pipeline, the candidate must not be worse than the current model on the same holdout by more than an agreed margin.
- **Behavioural tests:** check properties a sensible model must have. An *invariance* test changes something irrelevant (a customer's name) and expects the same prediction. A *directional* test changes something meaningful (many more support tickets) and expects churn risk to go up, not down.

```python
def test_more_tickets_raises_risk():
    base = {"tenure_months": 12, "monthly_spend": 50, "support_tickets": 0}
    worse = {**base, "support_tickets": 6}
    assert model_proba(worse) > model_proba(base)
```

## Worked example: a GitHub Actions workflow

GitHub Actions runs workflows defined in `.github/workflows/*.yml`. This one runs code, data and model tests on every push and pull request:

```yaml
name: ml-ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest -q tests/
```

Branch protection can then require this check to pass before a pull request is merged.

**Cost note:** GitHub Actions is free on public repositories with standard runners; private repositories on a free plan get a monthly allowance of minutes, and usage beyond it is billed. Keep model tests small and check the current terms on GitHub's billing page before relying on them.

## From CI to CD

Continuous *delivery* for ML usually means: when the pipeline produces a candidate that passes every gate, it is registered (for example in the MLflow Model Registry) and promoted step by step. A typical path is staging, then a **shadow** deployment (the new model scores real traffic but its answers are not used), then a **canary** (a small share of traffic, say 5%), then full rollout, with an automatic rollback if online metrics drop. Keep a human approval step for high-impact models.

## Common mistakes

- **Only testing code.** All green CI, broken model.
- **Flaky model tests.** Random seeds not fixed, or a threshold right at the metric's edge, so tests fail at random and people learn to ignore them.
- **Slow CI.** Training on the full dataset in every push. Use a small fixed sample in CI and the full data in the pipeline.
- **Thresholds without a baseline.** "Accuracy above 0.7" means nothing if predicting the majority class scores 0.9.

## Quick recap

- ML CI tests code, data and model quality, from fast to slow.
- Use fixed samples and seeds so model tests are fast and deterministic.
- Add behavioural tests (invariance, direction) alongside metric floors.
- CD promotes passing models through staging, shadow and canary with rollback; watch CI minutes on private repos.

## Try it

Open the **Do** tab and add data and model tests to a small project, then run them in a free GitHub Actions workflow.


Add code, data and model tests to continuous integration and promote models safely with shadow and canary releases.

CI/CD for ML

MLOps Explained - What It Is, Why You Need It and How It Works

Design retrainable training pipelines, prevent training/serving skew with shared features, and add data and model tests to CI/CD.