A mid-level course for machine learning engineers who can already train and evaluate models. You will make training reproducible by pinning code, data, environment and seeds; wrap a model in a validated API and package it in a container; and choose between batch, online and streaming inference. The pipelines module covers retrainable training pipelines, feature stores and training/serving skew, and CI/CD that tests code, data and model quality. The monitoring module teaches data and concept drift (you implement the Population Stability Index in NumPy), online metrics, SLOs, alerts and runbooks, and model governance: risk tiers, model cards, approvals and frameworks such as the NIST AI RMF. In-browser activities use plain Python and NumPy; tools such as Docker, MLflow, FastAPI and GitHub Actions are practised locally or on free tiers, with cost warnings and cleanup steps. Prerequisites: Applied Machine Learning, Data Structures & Algorithms and Cloud Fundamentals.

Take models from notebook to production: reproducible training, serving APIs, retrainable pipelines, CI/CD for ML, drift detection, monitoring and governance.

MLOps: Production Machine Learning

Create a new folder, run `git init`, create a virtual environment (`python -m venv .venv`) and install only what you need (for example numpy, pandas, scikit-learn, mlflow).

Move your training code out of any notebook into `train.py`, with hyperparameters and the data path in a `params.json` file.

Pin the environment: run `pip freeze > requirements.txt` (or use a lock-file tool) and commit it together with the code.

Add data pinning: compute the SHA-256 of the data file in `train.py` and stop with a clear error if it does not match the hash stored in `params.json`.

Fix every random seed you use (Python `random`, NumPy, and the library's `random_state`) from one `seed` value in `params.json`.

Log each run with MLflow: parameters, the git commit (`git rev-parse HEAD`), the data hash, the main metric and `requirements.txt` as an artefact. Start the UI with `mlflow ui` and open http://127.0.0.1:5000.

Train twice from a clean commit and compare the two runs in the MLflow UI: the metric should match exactly or within a tiny tolerance you can explain.

Optional: write a Dockerfile with a pinned Python base image, build it and run the training inside the container, then confirm the metric matches again.

Clean up: stop the MLflow UI, and remove any Docker images you no longer need with `docker image rm <name>` so they do not fill your disk.

## Why this matters

A model in production is a promise: "this artefact, trained on this data with this code, behaves like the one we evaluated." If you cannot rebuild it, you cannot debug it, audit it or safely improve it. A surprising number of production incidents start with a sentence like "it worked on my laptop" or "we are not sure which data the current model was trained on." Reproducible training turns a one-off notebook result into something a team can trust.

## The core idea: pin three things

Every trained model is a function of three inputs. Pin all three and you can rebuild the model.

1. **Code**: the exact commit of your training script, feature code and configuration.
2. **Data**: the exact snapshot of the training data (not "the customers table", but "the customers table as of 2026-10-01, hash `9f2c...`").
3. **Environment**: the Python version, every library version and, ideally, the operating system image.

Add a fourth, smaller item: **randomness**. Shuffling, weight initialisation and train/test splits all use random numbers, so fix the seeds.

## Pinning code

Commit everything that affects training, including configuration such as hyperparameters. Keep configuration in a file (`config.yaml` or `params.json`) rather than in notebook cells you edited by hand. When you train, record the commit hash:

```bash
git rev-parse HEAD
```

If the working tree has uncommitted changes, the hash alone does not describe the code. Many teams make the training script refuse to run on a "dirty" tree, or at least log a warning.

## Pinning data

Data is large and changes, so you usually do not commit it to Git directly. Common options:

- **Content hashes.** Compute a SHA-256 of the data file and store it with the model. If the hash changes, the data changed.
- **Data version control (DVC).** DVC stores small pointer files in Git and the real data in a cache or remote storage, so `git checkout` plus `dvc checkout` restores matching code and data.
- **Immutable snapshots.** Write each training set to a dated, read-only location (for example `s3://bucket/train/2026-10-01/`) and never overwrite it.

```python
import hashlib

def file_sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()
```

## Pinning the environment

`pip install pandas` gives you whatever version is current today. Six months later it gives you a different one. Pin exact versions:

```text
# requirements.txt
numpy==2.1.3
pandas==2.2.3
scikit-learn==1.5.2
```

`pip freeze > requirements.txt` captures what is installed; lock-file tools (pip-tools, Poetry, uv) do the same with dependency resolution. A **container image** goes further and pins the operating system and system libraries too:

```dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "train.py"]
```

Pin the base image as well (`python:3.12-slim` moves over time; a digest such as `python:3.12-slim@sha256:...` does not).

## Recording a run

Experiment trackers such as **MLflow** save the parameters, metrics, code version and artefacts of each run. MLflow runs locally with no account: `pip install mlflow`, log from your script, and browse runs with `mlflow ui` at `http://127.0.0.1:5000`.

```python
import mlflow

with mlflow.start_run():
    mlflow.log_params({"seed": 42, "max_depth": 6, "data_sha256": data_hash})
    mlflow.log_param("git_commit", commit)
    mlflow.log_metric("val_auc", auc)
    mlflow.log_artifact("requirements.txt")
```

## Worked example: same inputs, same model

You train a gradient-boosted model twice on Monday and get validation AUC 0.871 both times. On Thursday a colleague gets 0.864. Because each run logged the commit, data hash and package versions, the diff takes one minute: the commit and data match, but their environment had a newer scikit-learn whose default changed. Pinning the version restores 0.871. Without the records, that investigation could take days.

Exact bit-for-bit equality is not always possible (GPU kernels and multi-threading can add tiny non-determinism). The practical goal is: same inputs give the same model within a small, known tolerance, and you can explain any difference.

## Common mistakes

- **Training from a live table.** The data changes under you. Snapshot first, then train on the snapshot.
- **Unpinned dependencies.** "Latest" is not a version.
- **Seeds set in one place only.** Python's `random`, NumPy and your framework each have their own generator.
- **Hand-edited notebooks.** Cells run out of order cannot be replayed. Move training into a script.
- **Not saving the preprocessing.** The scaler, vocabulary or encoder is part of the model; version it with the model.

## Quick recap

- A model is a function of code, data, environment and seeds: pin all four.
- Use Git commits, data hashes or DVC, exact package versions and a container image.
- Log every run (parameters, metrics, commit, data hash) with a tracker such as MLflow.
- Aim for "same inputs, same model within tolerance" and be able to explain any drift.

## Try it

Open the **Do** tab and make a small training project fully reproducible on your own machine, then prove it by training twice.


Pin code, data, environment and random seeds so any model can be rebuilt and explained.

Reproducible training

Version Control for Data Science Explained in 5 Minutes (No Code!)

MLOps explained | Machine Learning Essentials

## Why this matters

A model that lives only in a notebook helps nobody. To be useful it has to answer requests from other software: a website asking "is this transaction fraud?", a mobile app asking for recommendations. The most common way to do that is to wrap the model in a small web API. Getting this layer right (input validation, versioning, error handling) matters as much as the model itself, because it is what other teams actually depend on.

## The core idea: a thin, strict wrapper

A model-serving API does four things on every request:

1. **Parse and validate** the input against a schema. Reject bad requests with a clear error instead of guessing.
2. **Transform** the input into the exact features the model was trained on, using the same preprocessing code.
3. **Predict** with a model that was loaded once at start-up, not on every request.
4. **Respond** with the prediction plus metadata such as the model version, so callers and logs know which model answered.

Keep business logic out of this layer. The API is a contract: "send me these fields, get back this shape."

## Worked example: FastAPI

FastAPI is a popular Python framework for this. It uses Pydantic models to validate requests automatically and returns HTTP 422 when a field is missing or has the wrong type.

```python
# app.py  (run locally: pip install fastapi uvicorn; uvicorn app:app --reload)
from fastapi import FastAPI
from pydantic import BaseModel, Field
import joblib

MODEL_VERSION = "churn-2026-10-01"
model = joblib.load("model.joblib")          # loaded once, at start-up

class Customer(BaseModel):
    tenure_months: float = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0)
    support_tickets: int = Field(ge=0)

app = FastAPI()

@app.get("/health")
def health():
    return {"status": "ok", "model_version": MODEL_VERSION}

@app.post("/predict")
def predict(c: Customer):
    x = [[c.tenure_months, c.monthly_spend, c.support_tickets]]
    p = float(model.predict_proba(x)[0, 1])
    return {"churn_probability": round(p, 4), "model_version": MODEL_VERSION}
```

A request and response look like this:

```bash
curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"tenure_months": 3, "monthly_spend": 80, "support_tickets": 4}'
# {"churn_probability": 0.7311, "model_version": "churn-2026-10-01"}
```

FastAPI also generates interactive documentation at `/docs`, which is a handy way to test by hand.

## Status codes that matter

| situation | status |
|---|---|
| valid request, prediction made | 200 |
| body is not valid JSON | 400 |
| JSON is valid but a field is missing, the wrong type or out of range | 422 |
| the model or a dependency failed | 500 (and log it) |

Never return a prediction for input you could not validate. A silently filled-in default (for example treating a missing `tenure_months` as 0) produces confident nonsense that nobody notices.

## Packaging for deployment

Put the API in a container image with pinned dependencies (as in the previous lesson), expose one port, and add the `/health` endpoint so the platform can check the service is alive. You can run the image locally with Docker (`docker build -t churn-api .` then `docker run -p 8000:8000 churn-api`). Managed platforms (cloud container services, serverless functions, or specialised model servers such as BentoML, TorchServe or KServe) run the same image at scale. If you try a cloud service, use its free tier, set a budget alert first, and delete the service when you finish: an idle endpoint can still bill by the hour.

## Performance basics

- **Load once.** Reading a model file on every request can add hundreds of milliseconds.
- **Batch when you can.** Many models predict 100 rows almost as fast as 1.
- **Measure latency percentiles** (p50, p95, p99), not just the average: the slowest requests are what users notice.

## Common mistakes

- **Training/serving mismatch.** The API rebuilds features with different code from training. Share one feature function between both.
- **No version in the response.** When something goes wrong you cannot tell which model answered.
- **Accepting anything.** Missing range checks let negative ages or 10-million-dollar invoices through.
- **Leaking internals.** Return a short error message, not a stack trace.

## Quick recap

- Validate, transform, predict, respond: keep the serving layer thin and strict.
- Load the model once and include the model version in every response.
- Use 422 for invalid input and never invent defaults for missing fields.
- Ship it as a container with a health check; watch costs on any cloud deployment.

## Try it

Open the **Do** tab and write the request handler for a small model API in plain Python: validation, prediction and the right status codes.


Wrap a trained model in a strict web API that validates input, returns versioned predictions and uses the right status codes.

Serving models

Deploy ML models with FastAPI, Docker, and Heroku | Tutorial

## Why this matters

"Deploy the model" can mean very different systems. A nightly job that scores ten million customers and writes the results to a table is nothing like an API that must answer a payment request in 50 milliseconds. Picking the wrong pattern either wastes money (a real-time service nobody needed) or fails users (yesterday's predictions for a decision that depends on the last five minutes). Choosing the inference pattern is one of the first design decisions in any ML system.

## The core idea: when is the prediction needed?

Ask two questions:

1. **When must the prediction exist?** Before anyone asks (precompute), or only at the moment of the request?
2. **What data does it need?** Data that is already known hours ahead, or data that only exists at request time (the basket contents, the text just typed, the current location)?

The answers point to one of three patterns.

### Batch (offline) inference

A scheduled job loads a large set of rows, scores them all and writes predictions to storage (a database table, a file, a cache). Applications then *look up* the stored prediction.

- **Good for:** churn scores, weekly demand forecasts, product recommendations refreshed nightly, risk scores for a monthly review.
- **Strengths:** simple, cheap per prediction, easy to retry, easy to inspect before anyone sees the results, can use big models and slow features.
- **Weaknesses:** predictions are stale between runs, and you compute scores for rows nobody will ever look at.

### Online (real-time) inference

A service keeps the model in memory and answers each request synchronously, usually through an API like the one in the previous lesson.

- **Good for:** fraud checks at checkout, search ranking, content moderation on upload, anything that depends on request-time data.
- **Strengths:** fresh, uses context that only exists now, scores only what is asked for.
- **Weaknesses:** strict latency budgets, the service must stay up around the clock, features must be available quickly at request time, and cost is driven by peak traffic.

### Streaming (near-real-time) inference

A consumer reads events from a stream (for example Kafka or a cloud queue), scores each event within seconds and writes the result onward. Nobody waits on the response, but results are much fresher than a nightly batch.

- **Good for:** sensor anomaly alerts, updating a user's recommendations after each click, flagging suspicious log-ins for review.

## Worked example: choosing for three products

| product need | pattern | why |
|---|---|---|
| Email 2 million customers a "you may like" list every Monday | batch | the list can be computed days ahead; latency does not matter |
| Approve or decline a card payment | online | decision depends on this transaction; the customer is waiting |
| Alert an engineer when a pump's vibration looks abnormal | streaming | events arrive continuously; a few seconds of delay is fine |

A useful hybrid is **precompute plus online adjustment**: batch-compute candidate recommendations overnight, then re-rank the top 50 online using what the user did in this session. You get most of the freshness for a fraction of the cost.

## Latency and cost, roughly

An online service has a **latency budget**, for example "p99 under 100 ms end to end." That budget covers network time, feature lookups and the model, so a model that takes 300 ms on a CPU is simply not an option there, however accurate it is. Batch jobs have a **throughput** goal instead ("score 10 million rows before 6 a.m."), so they can use larger models and cheaper, interruptible compute.

On cost: a batch job pays only while it runs. An online endpoint pays for every hour it is up, often with at least one instance always on, even at 3 a.m. with no traffic. Start with batch unless the product truly needs request-time data.

## Common mistakes

- **Real-time by default.** Building an always-on service for predictions that could have been computed last night.
- **Batch for request-time decisions.** Using yesterday's score when the deciding signal (the current basket, the new message) did not exist yesterday.
- **Ignoring feature availability.** A feature that takes a 20-second warehouse query cannot be used online, even if it helped offline accuracy.
- **No fallback.** If the online model times out, what does the product do? Define a safe default (a rule, a cached score or "send to manual review").

## Quick recap

- Batch: precompute on a schedule, look up later. Cheap and simple, but stale.
- Online: answer each request synchronously. Fresh, but bound by latency and always-on cost.
- Streaming: score events within seconds without a waiting caller.
- Decide by when the prediction is needed and what data it depends on; start simple.

## Try it

Open the **Do** tab and take the quiz to practise choosing a pattern for realistic product requirements.


Choose between batch, online and streaming inference from when a prediction is needed and what data it depends on.

Batch vs online inference

"Online" prediction vs "batch" prediction in machine learning

Make training reproducible, serve a model behind a validated API, and choose between batch, online and streaming inference.

Packaging Models

Write one sentence describing the model and the decision it supports, and how often its data changes.

Draw the pipeline as a DAG on paper or in a text file: at least ingest, validate, features, train, evaluate and register steps.

For each step, write its inputs, outputs (with a versioned path such as `runs/<run_id>/...`) and the check that must pass before the next step runs.

Define the evaluation gate: the metric, the baseline it must beat, and the maximum allowed drop compared with the current production model.

Choose a retraining trigger (schedule, performance, drift or data volume) and write down why it fits this model.

Implement the steps as functions in a `pipeline.py` script (or as stages in a `Makefile` or `dvc.yaml`), with all settings read from a config file.

Run it twice with the same config and confirm the second run either reuses or exactly reproduces the first run's outputs without overwriting them.

Break it on purpose (for example, remove a column from the input data) and confirm the validate step stops the run with a clear message.

Clean up: delete old run folders you do not need and stop any local services you started.

## Why this matters

Models go stale. Customer behaviour changes, new products appear, sensors are replaced. Sooner or later you must retrain, and if retraining means "someone re-runs a notebook by hand", it will happen late, inconsistently and with mistakes. A **training pipeline** turns training into a repeatable, automated process that anyone on the team (or a scheduler) can run, and that produces a model you can compare against the one in production before you ship it.

## The core idea: a graph of small, checkable steps

A pipeline is a directed acyclic graph (DAG) of steps. Each step has defined inputs and outputs, and runs only when its inputs are ready.

```text
ingest -> validate -> features -> train -> evaluate -> register
                                              \-> (fail) stop and alert
```

| step | input | output | what it checks |
|---|---|---|---|
| ingest | source tables, date range | raw snapshot + hash | row counts are plausible |
| validate | raw snapshot | validated data | schema, nulls, value ranges |
| features | validated data | feature table | same code as serving |
| train | features, config, seed | model artefact | training finished, loss sane |
| evaluate | model, held-out data | metrics report | beats a baseline and the current model |
| register | model + report | registry entry ("candidate") | a human or rule approves |

Small steps make failures easy to locate and let you re-run only what changed.

## Properties of a good pipeline

- **Parameterised.** The date range, data path and hyperparameters come from configuration, not from edits to the code.
- **Idempotent.** Running a step twice with the same inputs gives the same output and does not duplicate data. Write to a new versioned location instead of appending.
- **Versioned outputs.** Every artefact (data snapshot, features, model, report) is stored with an ID that links back to the code commit and inputs. This is the reproducibility lesson, automated.
- **Gated.** The pipeline does not deploy automatically just because training finished. The evaluate step compares the candidate to a baseline and to the production model on the same held-out data, and stops if it is worse.
- **Observable.** Each step logs its duration, row counts and key metrics so you can see where time and quality go.

## Worked example: a retraining pipeline as plain Python

You do not need a platform to start. A pipeline can be a script whose functions are the steps:

```python
# pipeline.py
import json, pathlib

def run(config):
    out = pathlib.Path("runs") / config["run_id"]
    out.mkdir(parents=True, exist_ok=False)          # new folder per run: idempotent
    raw = ingest(config["start"], config["end"])     # returns a DataFrame
    check_schema(raw)                                # raises on bad data
    X, y = build_features(raw)                       # shared with serving
    model = train(X, y, seed=config["seed"], **config["params"])
    report = evaluate(model, config["holdout_path"])
    (out / "report.json").write_text(json.dumps(report, indent=2))
    if report["auc"] < config["min_auc"]:
        raise SystemExit("candidate below threshold; not registering")
    save_model(model, out / "model.joblib")
```

When this grows, orchestrators take over the scheduling, retries and caching: **Airflow**, **Prefect** and **Dagster** for general data pipelines; **Kubeflow Pipelines**, **Vertex AI Pipelines**, **SageMaker Pipelines** or **TFX** for ML on managed or Kubernetes platforms; and lightweight options such as **DVC pipelines** (`dvc.yaml`) or a `Makefile` that rebuild only changed stages. The design (steps, inputs, outputs, gates) stays the same whichever tool you pick.

## When to retrain

| trigger | example | trade-off |
|---|---|---|
| schedule | every Sunday night | simple; may retrain when nothing changed |
| performance | live accuracy falls below a threshold | needs labels, which may arrive late |
| drift | input distribution shifts (module 3) | catches change early; can raise false alarms |
| new data volume | 100,000 new labelled rows | good for fast-growing products |

Most teams start with a schedule and add drift or performance triggers once monitoring is in place.

## Common mistakes

- **One giant step.** A three-hour script that fails at the end and must be re-run from scratch.
- **Silent data problems.** No validation step, so an empty or mis-joined table trains a broken model.
- **Auto-deploying.** Shipping every newly trained model without comparing it to the one already in production.
- **Different feature code.** Training builds features one way and serving another (more on this next lesson).
- **Overwriting outputs.** Re-runs replace yesterday's artefacts, so you cannot compare or roll back.

## Quick recap

- A pipeline is a DAG of small steps with defined inputs, outputs and checks.
- Make it parameterised, idempotent, versioned, gated and observable.
- Start with a script or `Makefile`; adopt an orchestrator when scheduling and retries get hard.
- Retrain on a schedule first, then on drift or performance signals.

## Try it

Open the **Do** tab and design (and optionally run locally) a retrainable pipeline for a small model on your own machine.


Design a retrainable training pipeline of small, versioned, gated steps from ingestion to model registration.

Training pipelines

Intro to Kubeflow Pipelines

How to build an ML pipeline with TFX

## Why this matters

A model can score 0.90 AUC offline and behave far worse in production, with no bug in the model at all. Very often the cause is that the features it sees in production are not quite the features it was trained on. This gap is called **training/serving skew**, and it is one of the most common and most expensive failures in applied ML because nothing crashes: the model just quietly makes worse decisions.

## The core idea: training/serving skew

Skew means the feature values the model receives at prediction time differ from what the same row would have produced in training. It has three usual causes.

1. **Two implementations.** Data scientists compute `avg_order_value_30d` in SQL for training; engineers re-implement it in Java for the API. One rounds, one includes refunds, and the values drift apart.
2. **Different data freshness.** Training uses a complete, cleaned warehouse table; serving uses a live table where today's orders are still arriving.
3. **Point-in-time leakage.** Training rows accidentally use information from *after* the moment the prediction would have been made. Offline accuracy looks great because the model is peeking at the future, which it can never do in production.

### Worked example: leakage by a careless join

You predict on 1 March whether a customer will churn in March. Your training query joins each customer to their *current* `total_support_tickets`, computed today in October. Customers who churned had a burst of tickets in March and April, so the feature is strongly predictive offline. In production on 1 March, those future tickets do not exist yet. The fix is a **point-in-time correct** join: for each training row, use only feature values with a timestamp at or before the prediction time.

```sql
-- point-in-time join: latest feature value at or before the label's cut-off
SELECT l.customer_id, l.cutoff_date, f.tickets_30d
FROM labels l
JOIN features f
  ON f.customer_id = l.customer_id
 AND f.as_of = (SELECT MAX(as_of) FROM features
                WHERE customer_id = l.customer_id AND as_of <= l.cutoff_date);
```

## What a feature store does

A **feature store** is a system for defining features once and serving the same values to training and to production. The main parts:

- **Feature definitions**: the transformation code, written once and registered with a name, owner and version (for example `customer.tickets_30d`).
- **Offline store**: the full history of feature values with timestamps, usually in a warehouse or files, used to build training sets with point-in-time joins.
- **Online store**: the latest value per entity in a fast key-value database, so an API can fetch features in a few milliseconds.
- **Materialisation**: jobs that compute features and write them to both stores from the same definition.
- **Registry and discovery**: a catalogue so another team can reuse `tickets_30d` instead of rebuilding it slightly differently.

Examples include the open-source **Feast** and **Hopsworks**, and managed stores in Databricks, Vertex AI and SageMaker.

```text
           feature definition (one piece of code)
                 /                         \
     offline store (history)        online store (latest)
          |                                |
  point-in-time training set        low-latency lookup in the API
```

## Do you need one?

Not always. A feature store earns its cost when several models share features, when online serving needs fresh precomputed aggregates, or when point-in-time correctness is hard to get right by hand. A single batch model can avoid skew more cheaply by **sharing one feature function** between the training pipeline and the scoring job, and by logging the features actually used at prediction time so you can compare them with training data.

## Catching skew

- **Log serving features.** Store the exact feature vector each prediction used.
- **Compare distributions.** Regularly compare logged serving features with the training set (the drift tests in module 3 work here too).
- **Replay test.** Take a sample of production requests, recompute their features with the training code, and check that the values match within a tolerance.

## Common mistakes

- **Rewriting feature logic for production** instead of reusing the training code.
- **Joining on "current" values** when building historical training data.
- **Ignoring defaults.** Training drops rows with missing values; serving fills them with 0. The model never saw zeros there.
- **Buying a platform first.** Adopting a feature store before you have a feature-sharing problem.

## Quick recap

- Training/serving skew: production feature values differ from training ones, so performance drops silently.
- Causes: two implementations, different freshness, and point-in-time leakage.
- Feature stores define features once and serve them to an offline store (history) and an online store (latest).
- Small teams can share one feature function and log serving features instead.

## Try it

Open the **Do** tab and take the quiz to check you can spot skew and leakage in realistic setups.


Explain training/serving skew and point-in-time leakage, and when a feature store helps prevent them.

Feature stores

What is a Feature Store for Machine Learning?

MFML 082 - The training-serving skew

Install pytest and create a `tests/` folder. Add one unit test for a feature function, including an edge case (zero, missing value or unseen category).

Save a small fixed sample of the data (a few hundred rows you are allowed to share) in `tests/data/` and write a data test that checks the column names, value ranges and label values.

Write a model quality test: train on the sample with a fixed seed and assert the metric is above a floor that beats a simple baseline (for example the majority class) by a margin.

Write one directional or invariance test (for example, more support tickets should not lower churn risk).

Run `pytest -q` locally until everything passes, then break the data or model on purpose and confirm a test fails.

Create `.github/workflows/ml-ci.yml` that checks out the code, sets up Python with pip caching, installs pinned requirements and runs pytest on push and pull request.

Push to a GitHub repository, open a pull request, and confirm the workflow runs and its status appears on the pull request.

Clean up: if you used a private repository, check the Actions usage on your billing page; delete the repository or disable the workflow if you no longer need it.

## Why this matters

In ordinary software, continuous integration (CI) runs tests on every change so bugs are caught before they ship. ML systems need the same safety net, plus more: a change can break the code, the data or the model's quality, and only the first of those raises an exception. CI/CD for ML adds automated checks for data and model behaviour so that a "small refactor" cannot quietly cost you five points of accuracy in production.

## The core idea: test code, data and model

Think of three layers of tests, from fast to slow.

| layer | examples | when it runs |
|---|---|---|
| **code** | unit tests for feature functions, the API's input validation, config parsing | every push, seconds |
| **data** | schema, null rates, value ranges, row counts on a fixed sample | every push (sample) and every pipeline run (full) |
| **model** | minimum metric on a fixed holdout, comparison with the current model, behavioural tests | on changes to training code or data; minutes |

### Code tests

Feature code is ordinary code. Test it like any other function, especially the edge cases that silently break models: empty inputs, missing values, time zones and unseen categories.

```python
def test_tenure_bucket_handles_zero():
    assert tenure_bucket(0) == "0-3m"
```

### Data tests

Assert the assumptions your model depends on. Tools such as Great Expectations or pandera make this declarative, but plain `assert` works.

```python
def test_training_sample_schema():
    df = load_sample("tests/data/sample.csv")
    assert list(df.columns) == ["tenure_months", "monthly_spend", "support_tickets", "churned"]
    assert df["monthly_spend"].between(0, 100_000).all()
    assert df["churned"].isin([0, 1]).all()
```

### Model tests

- **Quality gate:** train on a small fixed dataset with a fixed seed and assert the metric is above a floor (for example AUC at least 0.80). Keep it small so CI stays fast.
- **Regression against production:** in the full pipeline, the candidate must not be worse than the current model on the same holdout by more than an agreed margin.
- **Behavioural tests:** check properties a sensible model must have. An *invariance* test changes something irrelevant (a customer's name) and expects the same prediction. A *directional* test changes something meaningful (many more support tickets) and expects churn risk to go up, not down.

```python
def test_more_tickets_raises_risk():
    base = {"tenure_months": 12, "monthly_spend": 50, "support_tickets": 0}
    worse = {**base, "support_tickets": 6}
    assert model_proba(worse) > model_proba(base)
```

## Worked example: a GitHub Actions workflow

GitHub Actions runs workflows defined in `.github/workflows/*.yml`. This one runs code, data and model tests on every push and pull request:

```yaml
name: ml-ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest -q tests/
```

Branch protection can then require this check to pass before a pull request is merged.

**Cost note:** GitHub Actions is free on public repositories with standard runners; private repositories on a free plan get a monthly allowance of minutes, and usage beyond it is billed. Keep model tests small and check the current terms on GitHub's billing page before relying on them.

## From CI to CD

Continuous *delivery* for ML usually means: when the pipeline produces a candidate that passes every gate, it is registered (for example in the MLflow Model Registry) and promoted step by step. A typical path is staging, then a **shadow** deployment (the new model scores real traffic but its answers are not used), then a **canary** (a small share of traffic, say 5%), then full rollout, with an automatic rollback if online metrics drop. Keep a human approval step for high-impact models.

## Common mistakes

- **Only testing code.** All green CI, broken model.
- **Flaky model tests.** Random seeds not fixed, or a threshold right at the metric's edge, so tests fail at random and people learn to ignore them.
- **Slow CI.** Training on the full dataset in every push. Use a small fixed sample in CI and the full data in the pipeline.
- **Thresholds without a baseline.** "Accuracy above 0.7" means nothing if predicting the majority class scores 0.9.

## Quick recap

- ML CI tests code, data and model quality, from fast to slow.
- Use fixed samples and seeds so model tests are fast and deterministic.
- Add behavioural tests (invariance, direction) alongside metric floors.
- CD promotes passing models through staging, shadow and canary with rollback; watch CI minutes on private repos.

## Try it

Open the **Do** tab and add data and model tests to a small project, then run them in a free GitHub Actions workflow.


Add code, data and model tests to continuous integration and promote models safely with shadow and canary releases.

CI/CD for ML

MLOps Explained - What It Is, Why You Need It and How It Works

Design retrainable training pipelines, prevent training/serving skew with shared features, and add data and model tests to CI/CD.

Pipelines

## Why this matters

A model learns patterns from the past and is used on the future. When the world changes, its accuracy decays, usually without any error message. Labels that would reveal the drop often arrive weeks later (did the customer actually churn? was the loan repaid?). Drift detection watches the *inputs* and, where possible, the *relationship* between inputs and outcomes, so you notice change early instead of after the damage.

## The core idea: two kinds of drift

- **Data drift (covariate shift):** the distribution of the inputs `P(X)` changes. A redesigned app makes sessions longer; a marketing campaign brings in younger customers; a new sensor reads 2 degrees higher. The model may still be right for each kind of input, but it now sees inputs it saw rarely in training.
- **Concept drift:** the relationship between inputs and the target `P(y | X)` changes. The same transaction pattern that was safe last year is now a common fraud trick; during a recession, the same income predicts a higher default rate. The inputs can look identical while the right answer has changed.

There is also **label drift** (the share of positives changes) and plain **data quality** problems (a column suddenly all nulls, units switched from kg to g). Quality checks should run before any statistical drift test: a broken pipeline is not drift.

## Measuring data drift for one feature

You compare a **reference** sample (usually the training data or a known-good period) with a **current** window (for example the last 7 days of production).

### Population Stability Index (PSI)

PSI is widely used in industry, especially in credit risk:

1. Cut the reference data into bins, typically 10 bins at the reference deciles so each holds about 10% of the reference.
2. Compute the share of reference values `e_i` and current values `a_i` in each bin.
3. Sum the contributions:

```text
PSI = sum over bins of (a_i - e_i) * ln(a_i / e_i)
```

A tiny epsilon replaces empty-bin shares of 0 so the logarithm is defined.

**Worked example.** With two bins, the reference splits 50% / 50% and the current window splits 70% / 30%:

```text
(0.70 - 0.50) * ln(0.70 / 0.50) = 0.20 * 0.3365 = 0.0673
(0.30 - 0.50) * ln(0.30 / 0.50) = -0.20 * -0.5108 = 0.1022
PSI = 0.0673 + 0.1022 = 0.1695
```

A common rule of thumb reads PSI as: below 0.1 stable, 0.1 to 0.25 moderate shift (investigate), above 0.25 significant shift. These cut-offs are conventions, not laws: tune them for each feature using historical data.

### Other tests

- **Kolmogorov-Smirnov (KS):** the largest gap between two cumulative distributions, for numeric features (`scipy.stats.ks_2samp`).
- **Chi-squared test:** for categorical features, compares category counts.
- **Wasserstein distance / Jensen-Shannon divergence:** smooth distances that do not depend on a p-value.

Be careful with p-values at production volumes: with a million rows, a KS test will call a meaningless 0.1% shift "significant". Effect-size measures such as PSI or a distance with a tuned threshold usually make better alerts.

## Detecting concept drift

Concept drift needs outcomes. Options, from best to weakest:

1. **Live performance** once labels arrive: accuracy, AUC or error on recent predictions, compared with the validation result.
2. **Proxy signals** that arrive faster: chargebacks for fraud, returns for recommendations, manual review overturn rates.
3. **Prediction drift:** a shift in the distribution of the model's scores, which can hint at change before labels exist.

Libraries such as Evidently, NannyML and alibi-detect package these checks into reports and dashboards.

## Choosing windows

The reference and current windows shape what you detect. A 1-day window is noisy and catches weekday effects; a 90-day window reacts slowly. Many teams compare the last 7 days with the training set and also with the same week last year to separate seasonality from true change.

## Common mistakes

- **Alerting on every feature.** With 200 features, a few will always cross a threshold. Weight by feature importance and alert on the ones that matter.
- **Bins from the current data.** Bins must come from the reference, or the comparison is meaningless.
- **Treating drift as failure.** Drift says "look", not "retrain now". Check whether performance actually moved.
- **Ignoring seasonality.** December is not drift if every December looks like this.

## Quick recap

- Data drift: inputs change. Concept drift: the input-outcome relationship changes.
- PSI compares binned shares; under 0.1 stable, 0.1 to 0.25 moderate, over 0.25 significant (tune per feature).
- Run data-quality checks first; use effect sizes rather than raw p-values at scale.
- Confirm concept drift with live labels or proxies before retraining.

## Try it

Open the **Do** tab and implement PSI in NumPy to flag drift in a feature after an app redesign.


Tell data drift from concept drift and detect drift in a feature with the Population Stability Index.

Data and concept drift

Machine Learning Monitoring: What Is Data Drift?

1.1. ML lifecycle. What can go wrong with ML in production?

Describe the decision the model supports, who is affected when it is wrong, and how long it takes to learn the true outcome.

List 2 service-health metrics (for example error rate and p99 latency) with a target for each.

List 2 data-quality checks for the inputs most likely to break (null rate, value range, row count).

List 2 model-behaviour metrics (for example share of tickets marked urgent, PSI of the top features) with their normal range.

List 1-2 ground-truth or proxy metrics and how often you can compute them.

Write one SLO in the form "X% of ... over 30 days" for the most important metric.

Write two alerts: one that pages someone (user-facing breakage) and one that opens a ticket (slower ML signal), each with a threshold and a duration.

Write a short runbook for the paging alert: what it means, the first three checks, how to roll back to the previous model and who to escalate to.

Check for a feedback loop: does the model's action change the data it will be retrained on? Write how you would keep unbiased labels.

## Why this matters

Once a model is live, it is a production service like any other, and it can fail in all the usual ways (timeouts, crashes, a full disk) plus some new ones (drift, skew, a feedback loop). Monitoring is how you find out before your users or your finance team do. Good ML monitoring answers three questions continuously: *Is the service up? Is the data sane? Are the predictions still good?*

## The core idea: four layers of signals

| layer | example metrics | how fast you learn |
|---|---|---|
| **service health** | request rate, error rate (5xx), latency p50/p95/p99, CPU/memory, instance count | seconds |
| **data quality** | null rate per feature, out-of-range values, schema changes, row counts per batch | minutes |
| **model behaviour** | prediction distribution, share of positive predictions, feature drift (PSI), confidence | hours |
| **business / ground truth** | accuracy or AUC once labels arrive, conversion, fraud losses, manual review overturn rate | days to weeks |

The top layers are fast but indirect; the bottom layer is what you really care about but arrives late. You need all four.

## Choosing online metrics

Start from the decision the model supports and work backwards.

**Worked example: a fraud model at checkout.**

- *Service:* p99 latency under 80 ms (the checkout page waits on it); error rate under 0.1%.
- *Data:* null rate of `device_id` (a known fragile field) under 2%; transaction amount never negative.
- *Model behaviour:* share of transactions flagged stays within 0.5% to 2% (it has been about 1% for a year); PSI of the top five features under 0.25.
- *Ground truth:* precision and recall computed weekly from chargebacks, which arrive 30 to 60 days later; the manual review team's "not fraud" rate as a faster proxy.

Each metric gets an owner, a dashboard panel and, only if it is actionable, an alert.

## SLIs, SLOs and alerts

Borrow vocabulary from site reliability engineering:

- **SLI (service level indicator):** a measured value, such as "share of requests answered under 80 ms".
- **SLO (service level objective):** the target, such as "99.5% of requests under 80 ms over 30 days".
- **Alert:** a rule that pages or messages someone when an SLO is at risk.

Good alerts are **actionable** (someone knows what to do), **rare** (or people mute them) and **routed** (to the team that can fix it). Use two tiers: *page* for user-facing breakage (error rate spike, latency SLO burning fast) and *ticket* for slower ML signals (moderate drift, a dip in weekly precision).

```text
ALERT fraud_flag_rate_high
  IF flagged_share over 15 minutes > 3%   (normal: about 1%)
  FOR 15m
  SEVERITY page
  RUNBOOK https://wiki.example/runbooks/fraud-flag-rate
```

Every alert links to a **runbook**: what the alert means, the first three things to check, how to roll back to the previous model, and who to escalate to.

## Tooling

Service metrics usually go to **Prometheus** with **Grafana** dashboards, or to a cloud provider's monitoring service. Model and data metrics can be computed by scheduled jobs using libraries such as Evidently or whylogs, written to the same metrics store, and charted next to the service metrics. Log every prediction with its model version, input features and a request ID so you can later join it to the true outcome. Treat those logs as sensitive data: apply the same access controls and retention limits as the source data, and do not log more personal information than you need.

## Feedback loops

Some models change the data they will be retrained on. A fraud model that blocks a transaction never learns whether it was really fraud. A recommender only sees clicks on what it chose to show. Keep a small, randomised **holdout** (for example 1% of traffic scored but not acted on, where that is safe and legal) or human-reviewed samples, so you still get unbiased labels.

## Common mistakes

- **Only monitoring the service.** The API is green while the model has been returning near-constant scores for a week.
- **Alert floods.** Dozens of noisy drift alerts that everyone ignores; the one real incident is missed.
- **No baseline.** "Flag rate is 1.8%": is that high? Store and show the normal range.
- **No version in logs.** You cannot tell whether the problem started with the new model.
- **Waiting for labels.** Use proxies and prediction distributions while ground truth is in transit.

## Quick recap

- Monitor four layers: service health, data quality, model behaviour, and ground truth.
- Derive metrics from the decision the model supports; give each an owner.
- Write SLOs, alert only on actionable signals, tier alerts, and link every alert to a runbook.
- Log predictions with model version and inputs; watch for feedback loops.

## Try it

Open the **Do** tab and write a monitoring plan, with metrics, thresholds, alerts and a runbook, for a model you know.


Choose online metrics, SLOs and actionable alerts across service health, data quality, model behaviour and ground truth.

Model monitoring

1.2. What is ML monitoring and observability?

1.3. ML monitoring metrics. What exactly can you monitor?

## Why this matters

Models make or shape decisions about people: who gets a loan, which job applicant is shortlisted, which patient is flagged for follow-up. When those decisions go wrong, "the model did it" is not an acceptable answer to a customer, a regulator or a court. **Model governance** is the set of records, reviews and controls that let an organisation say what a model does, who approved it, what its known limits are and how it is watched. For a machine learning engineer, it is part of the job, not paperwork someone else handles.

## The core idea: risk-based controls

Not every model needs the same scrutiny. A model that sorts internal support tickets carries less risk than one that sets credit limits. Most frameworks scale controls by **impact** (how much harm a wrong decision can do, and to whom) and **autonomy** (does a human review the output, or does it act directly?).

A practical tiering:

| tier | example | typical controls |
|---|---|---|
| low | internal ticket routing, demand forecast for one store | model card, standard tests, owner on record |
| medium | marketing targeting, product recommendations | plus fairness checks, monitoring with alerts, sign-off from the product owner |
| high | credit, hiring, insurance pricing, medical triage | plus independent validation, legal and compliance review, human oversight, documented appeal route, periodic re-approval |

## Documenting a model

Two widely used formats:

- **Model card**: what the model is for (and *not* for), training data summary, evaluation results overall and per relevant subgroup, known limitations, ethical considerations, owner and version. You met these in Applied Machine Learning.
- **Datasheet for datasets**: where the data came from, how it was collected, consent and licensing, known gaps and biases, retention.

Add a **model risk record** for anything medium or high tier: the risk tier and why, the decision it supports, failure modes and their impact, mitigations, the monitoring plan, and the approvals with names and dates.

## Approvals and the model lifecycle

A governed lifecycle has explicit gates:

1. **Proposal:** use case, tier, data sources, legal basis for using personal data.
2. **Development:** experiments tracked; data lineage recorded.
3. **Validation:** for high-tier models, someone who did not build it reproduces the results, challenges assumptions and tests robustness and fairness.
4. **Approval:** named owners sign off (business owner, ML lead, and for high tier risk/compliance).
5. **Deployment:** the registry records which version is live and when it changed; rollback is tested.
6. **Monitoring and review:** scheduled re-reviews (for example yearly), plus a re-review after significant drift or a change of use.
7. **Retirement:** decommission cleanly, keep records for the required period.

A model registry (such as the MLflow Model Registry) supports this by storing versions, stages and annotations, but the approvals themselves are a human process.

## Fairness, explainability and oversight

- **Fairness:** compare error rates and outcomes across relevant groups (for example false positive rates by age band). There are several fairness definitions and they can conflict; document which one you chose and why.
- **Explainability:** be able to explain the main drivers of a decision at the level the audience needs. Global feature importance for reviewers; reason codes for an individual decision where required (credit decisions in many countries require reasons).
- **Human oversight:** decide where a person reviews outputs, can override them, and how a person affected by a decision can contest it.

## Frameworks and law

You do not need to memorise them, but you should know they exist and where to read them:

- **NIST AI Risk Management Framework (AI RMF 1.0)**: a voluntary US framework organised around Govern, Map, Measure and Manage. See [www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework) (checked on 2026-10-23).
- **SR 11-7**: US banking supervisory guidance on model risk management, long the reference for validation in finance.
- **EU AI Act**: risk-based EU regulation that places extra obligations on "high-risk" systems such as those used in hiring or credit.
- **Privacy law** such as GDPR, which covers personal data used in training and logging.

Requirements differ by country and sector and change over time. Work with your legal and compliance teams; this lesson is an orientation, not legal advice.

## Common mistakes

- **Documentation after launch.** Writing the model card from memory months later.
- **One-size-fits-all process.** Heavy review for every small model, so teams route around it.
- **Approval by the builder.** High-impact models need independent validation.
- **Ignoring changes of use.** A model approved for marketing quietly reused for credit decisions.

## Quick recap

- Governance scales controls with impact and autonomy: tier your models.
- Keep a model card, a datasheet and, for higher tiers, a risk record with named approvals.
- Gate the lifecycle: proposal, validation, approval, monitoring, periodic review, retirement.
- Know the main frameworks (NIST AI RMF, SR 11-7, EU AI Act) and involve legal and compliance.

## Try it

Open the **Do** tab and take the quiz to practise tiering models and choosing the right controls.


Tier models by risk and document their limits, approvals and oversight with model cards and risk records.

Governance and risk

Introduction to Responsible AI

Mastering AI Risk: NIST’s Risk Management Framework Explained

Detect data and concept drift, choose online metrics and alerts, and govern model risk with documentation and approvals.

MLOps: Production Machine Learning

About this course

Course content

Packaging Models

Pipelines

Monitoring

Where this course leads

Careers that use it