Take models from notebook to production: reproducible training, serving APIs, retrainable pipelines, CI/CD for ML, drift detection, monitoring and governance.

MLOps: Production Machine Learning

## Why this matters

A model learns patterns from the past and is used on the future. When the world changes, its accuracy decays, usually without any error message. Labels that would reveal the drop often arrive weeks later (did the customer actually churn? was the loan repaid?). Drift detection watches the *inputs* and, where possible, the *relationship* between inputs and outcomes, so you notice change early instead of after the damage.

## The core idea: two kinds of drift

- **Data drift (covariate shift):** the distribution of the inputs `P(X)` changes. A redesigned app makes sessions longer; a marketing campaign brings in younger customers; a new sensor reads 2 degrees higher. The model may still be right for each kind of input, but it now sees inputs it saw rarely in training.
- **Concept drift:** the relationship between inputs and the target `P(y | X)` changes. The same transaction pattern that was safe last year is now a common fraud trick; during a recession, the same income predicts a higher default rate. The inputs can look identical while the right answer has changed.

There is also **label drift** (the share of positives changes) and plain **data quality** problems (a column suddenly all nulls, units switched from kg to g). Quality checks should run before any statistical drift test: a broken pipeline is not drift.

## Measuring data drift for one feature

You compare a **reference** sample (usually the training data or a known-good period) with a **current** window (for example the last 7 days of production).

### Population Stability Index (PSI)

PSI is widely used in industry, especially in credit risk:

1. Cut the reference data into bins, typically 10 bins at the reference deciles so each holds about 10% of the reference.
2. Compute the share of reference values `e_i` and current values `a_i` in each bin.
3. Sum the contributions:

```text
PSI = sum over bins of (a_i - e_i) * ln(a_i / e_i)
```

A tiny epsilon replaces empty-bin shares of 0 so the logarithm is defined.

**Worked example.** With two bins, the reference splits 50% / 50% and the current window splits 70% / 30%:

```text
(0.70 - 0.50) * ln(0.70 / 0.50) = 0.20 * 0.3365 = 0.0673
(0.30 - 0.50) * ln(0.30 / 0.50) = -0.20 * -0.5108 = 0.1022
PSI = 0.0673 + 0.1022 = 0.1695
```

A common rule of thumb reads PSI as: below 0.1 stable, 0.1 to 0.25 moderate shift (investigate), above 0.25 significant shift. These cut-offs are conventions, not laws: tune them for each feature using historical data.

### Other tests

- **Kolmogorov-Smirnov (KS):** the largest gap between two cumulative distributions, for numeric features (`scipy.stats.ks_2samp`).
- **Chi-squared test:** for categorical features, compares category counts.
- **Wasserstein distance / Jensen-Shannon divergence:** smooth distances that do not depend on a p-value.

Be careful with p-values at production volumes: with a million rows, a KS test will call a meaningless 0.1% shift "significant". Effect-size measures such as PSI or a distance with a tuned threshold usually make better alerts.

## Detecting concept drift

Concept drift needs outcomes. Options, from best to weakest:

1. **Live performance** once labels arrive: accuracy, AUC or error on recent predictions, compared with the validation result.
2. **Proxy signals** that arrive faster: chargebacks for fraud, returns for recommendations, manual review overturn rates.
3. **Prediction drift:** a shift in the distribution of the model's scores, which can hint at change before labels exist.

Libraries such as Evidently, NannyML and alibi-detect package these checks into reports and dashboards.

## Choosing windows

The reference and current windows shape what you detect. A 1-day window is noisy and catches weekday effects; a 90-day window reacts slowly. Many teams compare the last 7 days with the training set and also with the same week last year to separate seasonality from true change.

## Common mistakes

- **Alerting on every feature.** With 200 features, a few will always cross a threshold. Weight by feature importance and alert on the ones that matter.
- **Bins from the current data.** Bins must come from the reference, or the comparison is meaningless.
- **Treating drift as failure.** Drift says "look", not "retrain now". Check whether performance actually moved.
- **Ignoring seasonality.** December is not drift if every December looks like this.

## Quick recap

- Data drift: inputs change. Concept drift: the input-outcome relationship changes.
- PSI compares binned shares; under 0.1 stable, 0.1 to 0.25 moderate, over 0.25 significant (tune per feature).
- Run data-quality checks first; use effect sizes rather than raw p-values at scale.
- Confirm concept drift with live labels or proxies before retraining.

## Try it

Open the **Do** tab and implement PSI in NumPy to flag drift in a feature after an app redesign.


Tell data drift from concept drift and detect drift in a feature with the Population Stability Index.

Data and concept drift

Machine Learning Monitoring: What Is Data Drift?

1.1. ML lifecycle. What can go wrong with ML in production?

Describe the decision the model supports, who is affected when it is wrong, and how long it takes to learn the true outcome.

List 2 service-health metrics (for example error rate and p99 latency) with a target for each.

List 2 data-quality checks for the inputs most likely to break (null rate, value range, row count).

List 2 model-behaviour metrics (for example share of tickets marked urgent, PSI of the top features) with their normal range.

List 1-2 ground-truth or proxy metrics and how often you can compute them.

Write one SLO in the form "X% of ... over 30 days" for the most important metric.

Write two alerts: one that pages someone (user-facing breakage) and one that opens a ticket (slower ML signal), each with a threshold and a duration.

Write a short runbook for the paging alert: what it means, the first three checks, how to roll back to the previous model and who to escalate to.

Check for a feedback loop: does the model's action change the data it will be retrained on? Write how you would keep unbiased labels.

## Why this matters

Once a model is live, it is a production service like any other, and it can fail in all the usual ways (timeouts, crashes, a full disk) plus some new ones (drift, skew, a feedback loop). Monitoring is how you find out before your users or your finance team do. Good ML monitoring answers three questions continuously: *Is the service up? Is the data sane? Are the predictions still good?*

## The core idea: four layers of signals

| layer | example metrics | how fast you learn |
|---|---|---|
| **service health** | request rate, error rate (5xx), latency p50/p95/p99, CPU/memory, instance count | seconds |
| **data quality** | null rate per feature, out-of-range values, schema changes, row counts per batch | minutes |
| **model behaviour** | prediction distribution, share of positive predictions, feature drift (PSI), confidence | hours |
| **business / ground truth** | accuracy or AUC once labels arrive, conversion, fraud losses, manual review overturn rate | days to weeks |

The top layers are fast but indirect; the bottom layer is what you really care about but arrives late. You need all four.

## Choosing online metrics

Start from the decision the model supports and work backwards.

**Worked example: a fraud model at checkout.**

- *Service:* p99 latency under 80 ms (the checkout page waits on it); error rate under 0.1%.
- *Data:* null rate of `device_id` (a known fragile field) under 2%; transaction amount never negative.
- *Model behaviour:* share of transactions flagged stays within 0.5% to 2% (it has been about 1% for a year); PSI of the top five features under 0.25.
- *Ground truth:* precision and recall computed weekly from chargebacks, which arrive 30 to 60 days later; the manual review team's "not fraud" rate as a faster proxy.

Each metric gets an owner, a dashboard panel and, only if it is actionable, an alert.

## SLIs, SLOs and alerts

Borrow vocabulary from site reliability engineering:

- **SLI (service level indicator):** a measured value, such as "share of requests answered under 80 ms".
- **SLO (service level objective):** the target, such as "99.5% of requests under 80 ms over 30 days".
- **Alert:** a rule that pages or messages someone when an SLO is at risk.

Good alerts are **actionable** (someone knows what to do), **rare** (or people mute them) and **routed** (to the team that can fix it). Use two tiers: *page* for user-facing breakage (error rate spike, latency SLO burning fast) and *ticket* for slower ML signals (moderate drift, a dip in weekly precision).

```text
ALERT fraud_flag_rate_high
  IF flagged_share over 15 minutes > 3%   (normal: about 1%)
  FOR 15m
  SEVERITY page
  RUNBOOK https://wiki.example/runbooks/fraud-flag-rate
```

Every alert links to a **runbook**: what the alert means, the first three things to check, how to roll back to the previous model, and who to escalate to.

## Tooling

Service metrics usually go to **Prometheus** with **Grafana** dashboards, or to a cloud provider's monitoring service. Model and data metrics can be computed by scheduled jobs using libraries such as Evidently or whylogs, written to the same metrics store, and charted next to the service metrics. Log every prediction with its model version, input features and a request ID so you can later join it to the true outcome. Treat those logs as sensitive data: apply the same access controls and retention limits as the source data, and do not log more personal information than you need.

## Feedback loops

Some models change the data they will be retrained on. A fraud model that blocks a transaction never learns whether it was really fraud. A recommender only sees clicks on what it chose to show. Keep a small, randomised **holdout** (for example 1% of traffic scored but not acted on, where that is safe and legal) or human-reviewed samples, so you still get unbiased labels.

## Common mistakes

- **Only monitoring the service.** The API is green while the model has been returning near-constant scores for a week.
- **Alert floods.** Dozens of noisy drift alerts that everyone ignores; the one real incident is missed.
- **No baseline.** "Flag rate is 1.8%": is that high? Store and show the normal range.
- **No version in logs.** You cannot tell whether the problem started with the new model.
- **Waiting for labels.** Use proxies and prediction distributions while ground truth is in transit.

## Quick recap

- Monitor four layers: service health, data quality, model behaviour, and ground truth.
- Derive metrics from the decision the model supports; give each an owner.
- Write SLOs, alert only on actionable signals, tier alerts, and link every alert to a runbook.
- Log predictions with model version and inputs; watch for feedback loops.

## Try it

Open the **Do** tab and write a monitoring plan, with metrics, thresholds, alerts and a runbook, for a model you know.


Choose online metrics, SLOs and actionable alerts across service health, data quality, model behaviour and ground truth.

Model monitoring

1.2. What is ML monitoring and observability?

1.3. ML monitoring metrics. What exactly can you monitor?

## Why this matters

Models make or shape decisions about people: who gets a loan, which job applicant is shortlisted, which patient is flagged for follow-up. When those decisions go wrong, "the model did it" is not an acceptable answer to a customer, a regulator or a court. **Model governance** is the set of records, reviews and controls that let an organisation say what a model does, who approved it, what its known limits are and how it is watched. For a machine learning engineer, it is part of the job, not paperwork someone else handles.

## The core idea: risk-based controls

Not every model needs the same scrutiny. A model that sorts internal support tickets carries less risk than one that sets credit limits. Most frameworks scale controls by **impact** (how much harm a wrong decision can do, and to whom) and **autonomy** (does a human review the output, or does it act directly?).

A practical tiering:

| tier | example | typical controls |
|---|---|---|
| low | internal ticket routing, demand forecast for one store | model card, standard tests, owner on record |
| medium | marketing targeting, product recommendations | plus fairness checks, monitoring with alerts, sign-off from the product owner |
| high | credit, hiring, insurance pricing, medical triage | plus independent validation, legal and compliance review, human oversight, documented appeal route, periodic re-approval |

## Documenting a model

Two widely used formats:

- **Model card**: what the model is for (and *not* for), training data summary, evaluation results overall and per relevant subgroup, known limitations, ethical considerations, owner and version. You met these in Applied Machine Learning.
- **Datasheet for datasets**: where the data came from, how it was collected, consent and licensing, known gaps and biases, retention.

Add a **model risk record** for anything medium or high tier: the risk tier and why, the decision it supports, failure modes and their impact, mitigations, the monitoring plan, and the approvals with names and dates.

## Approvals and the model lifecycle

A governed lifecycle has explicit gates:

1. **Proposal:** use case, tier, data sources, legal basis for using personal data.
2. **Development:** experiments tracked; data lineage recorded.
3. **Validation:** for high-tier models, someone who did not build it reproduces the results, challenges assumptions and tests robustness and fairness.
4. **Approval:** named owners sign off (business owner, ML lead, and for high tier risk/compliance).
5. **Deployment:** the registry records which version is live and when it changed; rollback is tested.
6. **Monitoring and review:** scheduled re-reviews (for example yearly), plus a re-review after significant drift or a change of use.
7. **Retirement:** decommission cleanly, keep records for the required period.

A model registry (such as the MLflow Model Registry) supports this by storing versions, stages and annotations, but the approvals themselves are a human process.

## Fairness, explainability and oversight

- **Fairness:** compare error rates and outcomes across relevant groups (for example false positive rates by age band). There are several fairness definitions and they can conflict; document which one you chose and why.
- **Explainability:** be able to explain the main drivers of a decision at the level the audience needs. Global feature importance for reviewers; reason codes for an individual decision where required (credit decisions in many countries require reasons).
- **Human oversight:** decide where a person reviews outputs, can override them, and how a person affected by a decision can contest it.

## Frameworks and law

You do not need to memorise them, but you should know they exist and where to read them:

- **NIST AI Risk Management Framework (AI RMF 1.0)**: a voluntary US framework organised around Govern, Map, Measure and Manage. See [www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework) (checked on 2026-10-23).
- **SR 11-7**: US banking supervisory guidance on model risk management, long the reference for validation in finance.
- **EU AI Act**: risk-based EU regulation that places extra obligations on "high-risk" systems such as those used in hiring or credit.
- **Privacy law** such as GDPR, which covers personal data used in training and logging.

Requirements differ by country and sector and change over time. Work with your legal and compliance teams; this lesson is an orientation, not legal advice.

## Common mistakes

- **Documentation after launch.** Writing the model card from memory months later.
- **One-size-fits-all process.** Heavy review for every small model, so teams route around it.
- **Approval by the builder.** High-impact models need independent validation.
- **Ignoring changes of use.** A model approved for marketing quietly reused for credit decisions.

## Quick recap

- Governance scales controls with impact and autonomy: tier your models.
- Keep a model card, a datasheet and, for higher tiers, a risk record with named approvals.
- Gate the lifecycle: proposal, validation, approval, monitoring, periodic review, retirement.
- Know the main frameworks (NIST AI RMF, SR 11-7, EU AI Act) and involve legal and compliance.

## Try it

Open the **Do** tab and take the quiz to practise tiering models and choosing the right controls.


Tier models by risk and document their limits, approvals and oversight with model cards and risk records.

Governance and risk

Introduction to Responsible AI

Mastering AI Risk: NIST’s Risk Management Framework Explained

Detect data and concept drift, choose online metrics and alerts, and govern model risk with documentation and approvals.