Choose between batch, online and streaming inference from when a prediction is needed and what data it depends on.
Why this matters
"Deploy the model" can mean very different systems. A nightly job that scores ten million customers and writes the results to a table is nothing like an API that must answer a payment request in 50 milliseconds. Picking the wrong pattern either wastes money (a real-time service nobody needed) or fails users (yesterday's predictions for a decision that depends on the last five minutes). Choosing the inference pattern is one of the first design decisions in any ML system.
The core idea: when is the prediction needed?
Ask two questions:
- When must the prediction exist? Before anyone asks (precompute), or only at the moment of the request?
- What data does it need? Data that is already known hours ahead, or data that only exists at request time (the basket contents, the text just typed, the current location)?
The answers point to one of three patterns.
Batch (offline) inference
A scheduled job loads a large set of rows, scores them all and writes predictions to storage (a database table, a file, a cache). Applications then look up the stored prediction.
- Good for: churn scores, weekly demand forecasts, product recommendations refreshed nightly, risk scores for a monthly review.
- Strengths: simple, cheap per prediction, easy to retry, easy to inspect before anyone sees the results, can use big models and slow features.
- Weaknesses: predictions are stale between runs, and you compute scores for rows nobody will ever look at.
Online (real-time) inference
A service keeps the model in memory and answers each request synchronously, usually through an API like the one in the previous lesson.
- Good for: fraud checks at checkout, search ranking, content moderation on upload, anything that depends on request-time data.
- Strengths: fresh, uses context that only exists now, scores only what is asked for.
- Weaknesses: strict latency budgets, the service must stay up around the clock, features must be available quickly at request time, and cost is driven by peak traffic.
Streaming (near-real-time) inference
A consumer reads events from a stream (for example Kafka or a cloud queue), scores each event within seconds and writes the result onward. Nobody waits on the response, but results are much fresher than a nightly batch.
- Good for: sensor anomaly alerts, updating a user's recommendations after each click, flagging suspicious log-ins for review.
Worked example: choosing for three products
| product need | pattern | why |
|---|---|---|
| Email 2 million customers a "you may like" list every Monday | batch | the list can be computed days ahead; latency does not matter |
| Approve or decline a card payment | online | decision depends on this transaction; the customer is waiting |
| Alert an engineer when a pump's vibration looks abnormal | streaming | events arrive continuously; a few seconds of delay is fine |
A useful hybrid is precompute plus online adjustment: batch-compute candidate recommendations overnight, then re-rank the top 50 online using what the user did in this session. You get most of the freshness for a fraction of the cost.
Latency and cost, roughly
An online service has a latency budget, for example "p99 under 100 ms end to end." That budget covers network time, feature lookups and the model, so a model that takes 300 ms on a CPU is simply not an option there, however accurate it is. Batch jobs have a throughput goal instead ("score 10 million rows before 6 a.m."), so they can use larger models and cheaper, interruptible compute.
On cost: a batch job pays only while it runs. An online endpoint pays for every hour it is up, often with at least one instance always on, even at 3 a.m. with no traffic. Start with batch unless the product truly needs request-time data.
Common mistakes
- Real-time by default. Building an always-on service for predictions that could have been computed last night.
- Batch for request-time decisions. Using yesterday's score when the deciding signal (the current basket, the new message) did not exist yesterday.
- Ignoring feature availability. A feature that takes a 20-second warehouse query cannot be used online, even if it helped offline accuracy.
- No fallback. If the online model times out, what does the product do? Define a safe default (a rule, a cached score or "send to manual review").
Quick recap
- Batch: precompute on a schedule, look up later. Cheap and simple, but stale.
- Online: answer each request synchronously. Fresh, but bound by latency and always-on cost.
- Streaming: score events within seconds without a waiting caller.
- Decide by when the prediction is needed and what data it depends on; start simple.
Try it
Open the Do tab and take the quiz to practise choosing a pattern for realistic product requirements.