Choose online metrics, SLOs and actionable alerts across service health, data quality, model behaviour and ground truth.
Why this matters
Once a model is live, it is a production service like any other, and it can fail in all the usual ways (timeouts, crashes, a full disk) plus some new ones (drift, skew, a feedback loop). Monitoring is how you find out before your users or your finance team do. Good ML monitoring answers three questions continuously: Is the service up? Is the data sane? Are the predictions still good?
The core idea: four layers of signals
| layer | example metrics | how fast you learn |
|---|---|---|
| service health | request rate, error rate (5xx), latency p50/p95/p99, CPU/memory, instance count | seconds |
| data quality | null rate per feature, out-of-range values, schema changes, row counts per batch | minutes |
| model behaviour | prediction distribution, share of positive predictions, feature drift (PSI), confidence | hours |
| business / ground truth | accuracy or AUC once labels arrive, conversion, fraud losses, manual review overturn rate | days to weeks |
The top layers are fast but indirect; the bottom layer is what you really care about but arrives late. You need all four.
Choosing online metrics
Start from the decision the model supports and work backwards.
Worked example: a fraud model at checkout.
- Service: p99 latency under 80 ms (the checkout page waits on it); error rate under 0.1%.
- Data: null rate of
device_id(a known fragile field) under 2%; transaction amount never negative. - Model behaviour: share of transactions flagged stays within 0.5% to 2% (it has been about 1% for a year); PSI of the top five features under 0.25.
- Ground truth: precision and recall computed weekly from chargebacks, which arrive 30 to 60 days later; the manual review team's "not fraud" rate as a faster proxy.
Each metric gets an owner, a dashboard panel and, only if it is actionable, an alert.
SLIs, SLOs and alerts
Borrow vocabulary from site reliability engineering:
- SLI (service level indicator): a measured value, such as "share of requests answered under 80 ms".
- SLO (service level objective): the target, such as "99.5% of requests under 80 ms over 30 days".
- Alert: a rule that pages or messages someone when an SLO is at risk.
Good alerts are actionable (someone knows what to do), rare (or people mute them) and routed (to the team that can fix it). Use two tiers: page for user-facing breakage (error rate spike, latency SLO burning fast) and ticket for slower ML signals (moderate drift, a dip in weekly precision).
ALERT fraud_flag_rate_high
IF flagged_share over 15 minutes > 3% (normal: about 1%)
FOR 15m
SEVERITY page
RUNBOOK https://wiki.example/runbooks/fraud-flag-rate
Every alert links to a runbook: what the alert means, the first three things to check, how to roll back to the previous model, and who to escalate to.
Tooling
Service metrics usually go to Prometheus with Grafana dashboards, or to a cloud provider's monitoring service. Model and data metrics can be computed by scheduled jobs using libraries such as Evidently or whylogs, written to the same metrics store, and charted next to the service metrics. Log every prediction with its model version, input features and a request ID so you can later join it to the true outcome. Treat those logs as sensitive data: apply the same access controls and retention limits as the source data, and do not log more personal information than you need.
Feedback loops
Some models change the data they will be retrained on. A fraud model that blocks a transaction never learns whether it was really fraud. A recommender only sees clicks on what it chose to show. Keep a small, randomised holdout (for example 1% of traffic scored but not acted on, where that is safe and legal) or human-reviewed samples, so you still get unbiased labels.
Common mistakes
- Only monitoring the service. The API is green while the model has been returning near-constant scores for a week.
- Alert floods. Dozens of noisy drift alerts that everyone ignores; the one real incident is missed.
- No baseline. "Flag rate is 1.8%": is that high? Store and show the normal range.
- No version in logs. You cannot tell whether the problem started with the new model.
- Waiting for labels. Use proxies and prediction distributions while ground truth is in transit.
Quick recap
- Monitor four layers: service health, data quality, model behaviour, and ground truth.
- Derive metrics from the decision the model supports; give each an owner.
- Write SLOs, alert only on actionable signals, tier alerts, and link every alert to a runbook.
- Log predictions with model version and inputs; watch for feedback loops.
Try it
Open the Do tab and write a monitoring plan, with metrics, thresholds, alerts and a runbook, for a model you know.