CI/CD for ML

Why this matters

In ordinary software, continuous integration (CI) runs tests on every change so bugs are caught before they ship. ML systems need the same safety net, plus more: a change can break the code, the data or the model's quality, and only the first of those raises an exception. CI/CD for ML adds automated checks for data and model behaviour so that a "small refactor" cannot quietly cost you five points of accuracy in production.

The core idea: test code, data and model

Think of three layers of tests, from fast to slow.

Code tests

Feature code is ordinary code. Test it like any other function, especially the edge cases that silently break models: empty inputs, missing values, time zones and unseen categories.

def test_tenure_bucket_handles_zero():
    assert tenure_bucket(0) == "0-3m"

Data tests

Assert the assumptions your model depends on. Tools such as Great Expectations or pandera make this declarative, but plain assert works.

def test_training_sample_schema():
    df = load_sample("tests/data/sample.csv")
    assert list(df.columns) == ["tenure_months", "monthly_spend", "support_tickets", "churned"]
    assert df["monthly_spend"].between(0, 100_000).all()
    assert df["churned"].isin([0, 1]).all()

Model tests

• Quality gate: train on a small fixed dataset with a fixed seed and assert the metric is above a floor (for example AUC at least 0.80). Keep it small so CI stays fast.
• Regression against production: in the full pipeline, the candidate must not be worse than the current model on the same holdout by more than an agreed margin.
• Behavioural tests: check properties a sensible model must have. An invariance test changes something irrelevant (a customer's name) and expects the same prediction. A directional test changes something meaningful (many more support tickets) and expects churn risk to go up, not down.

def test_more_tickets_raises_risk():
    base = {"tenure_months": 12, "monthly_spend": 50, "support_tickets": 0}
    worse = {**base, "support_tickets": 6}
    assert model_proba(worse) > model_proba(base)

Worked example: a GitHub Actions workflow

GitHub Actions runs workflows defined in .github/workflows/*.yml. This one runs code, data and model tests on every push and pull request:

name: ml-ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest -q tests/

Branch protection can then require this check to pass before a pull request is merged.

Cost note: GitHub Actions is free on public repositories with standard runners; private repositories on a free plan get a monthly allowance of minutes, and usage beyond it is billed. Keep model tests small and check the current terms on GitHub's billing page before relying on them.

From CI to CD

Continuous delivery for ML usually means: when the pipeline produces a candidate that passes every gate, it is registered (for example in the MLflow Model Registry) and promoted step by step. A typical path is staging, then a shadow deployment (the new model scores real traffic but its answers are not used), then a canary (a small share of traffic, say 5%), then full rollout, with an automatic rollback if online metrics drop. Keep a human approval step for high-impact models.

Common mistakes

• Only testing code. All green CI, broken model.
• Flaky model tests. Random seeds not fixed, or a threshold right at the metric's edge, so tests fail at random and people learn to ignore them.
• Slow CI. Training on the full dataset in every push. Use a small fixed sample in CI and the full data in the pipeline.
• Thresholds without a baseline. "Accuracy above 0.7" means nothing if predicting the majority class scores 0.9.

Quick recap

• ML CI tests code, data and model quality, from fast to slow.
• Use fixed samples and seeds so model tests are fast and deterministic.
• Add behavioural tests (invariance, direction) alongside metric floors.
• CD promotes passing models through staging, shadow and canary with rollback; watch CI minutes on private repos.

Try it

Open the Do tab and add data and model tests to a small project, then run them in a free GitHub Actions workflow.