Serving models

Why this matters

A model that lives only in a notebook helps nobody. To be useful it has to answer requests from other software: a website asking "is this transaction fraud?", a mobile app asking for recommendations. The most common way to do that is to wrap the model in a small web API. Getting this layer right (input validation, versioning, error handling) matters as much as the model itself, because it is what other teams actually depend on.

The core idea: a thin, strict wrapper

A model-serving API does four things on every request:

1. Parse and validate the input against a schema. Reject bad requests with a clear error instead of guessing.
2. Transform the input into the exact features the model was trained on, using the same preprocessing code.
3. Predict with a model that was loaded once at start-up, not on every request.
4. Respond with the prediction plus metadata such as the model version, so callers and logs know which model answered.

Keep business logic out of this layer. The API is a contract: "send me these fields, get back this shape."

Worked example: FastAPI

FastAPI is a popular Python framework for this. It uses Pydantic models to validate requests automatically and returns HTTP 422 when a field is missing or has the wrong type.

# app.py  (run locally: pip install fastapi uvicorn; uvicorn app:app --reload)
from fastapi import FastAPI
from pydantic import BaseModel, Field
import joblib

MODEL_VERSION = "churn-2026-10-01"
model = joblib.load("model.joblib")          # loaded once, at start-up

class Customer(BaseModel):
    tenure_months: float = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0)
    support_tickets: int = Field(ge=0)

app = FastAPI()

@app.get("/health")
def health():
    return {"status": "ok", "model_version": MODEL_VERSION}

@app.post("/predict")
def predict(c: Customer):
    x = [[c.tenure_months, c.monthly_spend, c.support_tickets]]
    p = float(model.predict_proba(x)[0, 1])
    return {"churn_probability": round(p, 4), "model_version": MODEL_VERSION}

A request and response look like this:

curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"tenure_months": 3, "monthly_spend": 80, "support_tickets": 4}'
# {"churn_probability": 0.7311, "model_version": "churn-2026-10-01"}

FastAPI also generates interactive documentation at /docs, which is a handy way to test by hand.

Status codes that matter

Never return a prediction for input you could not validate. A silently filled-in default (for example treating a missing tenure_months as 0) produces confident nonsense that nobody notices.

Packaging for deployment

Put the API in a container image with pinned dependencies (as in the previous lesson), expose one port, and add the /health endpoint so the platform can check the service is alive. You can run the image locally with Docker (docker build -t churn-api . then docker run -p 8000:8000 churn-api). Managed platforms (cloud container services, serverless functions, or specialised model servers such as BentoML, TorchServe or KServe) run the same image at scale. If you try a cloud service, use its free tier, set a budget alert first, and delete the service when you finish: an idle endpoint can still bill by the hour.

Performance basics

• Load once. Reading a model file on every request can add hundreds of milliseconds.
• Batch when you can. Many models predict 100 rows almost as fast as 1.
• Measure latency percentiles (p50, p95, p99), not just the average: the slowest requests are what users notice.

Common mistakes

• Training/serving mismatch. The API rebuilds features with different code from training. Share one feature function between both.
• No version in the response. When something goes wrong you cannot tell which model answered.
• Accepting anything. Missing range checks let negative ages or 10-million-dollar invoices through.
• Leaking internals. Return a short error message, not a stack trace.

Quick recap

• Validate, transform, predict, respond: keep the serving layer thin and strict.
• Load the model once and include the model version in every response.
• Use 422 for invalid input and never invent defaults for missing fields.
• Ship it as a container with a health check; watch costs on any cloud deployment.

Try it

Open the Do tab and write the request handler for a small model API in plain Python: validation, prediction and the right status codes.