Feature stores

Why this matters

A model can score 0.90 AUC offline and behave far worse in production, with no bug in the model at all. Very often the cause is that the features it sees in production are not quite the features it was trained on. This gap is called training/serving skew, and it is one of the most common and most expensive failures in applied ML because nothing crashes: the model just quietly makes worse decisions.

The core idea: training/serving skew

Skew means the feature values the model receives at prediction time differ from what the same row would have produced in training. It has three usual causes.

1. Two implementations. Data scientists compute avg_order_value_30d in SQL for training; engineers re-implement it in Java for the API. One rounds, one includes refunds, and the values drift apart.
2. Different data freshness. Training uses a complete, cleaned warehouse table; serving uses a live table where today's orders are still arriving.
3. Point-in-time leakage. Training rows accidentally use information from after the moment the prediction would have been made. Offline accuracy looks great because the model is peeking at the future, which it can never do in production.

Worked example: leakage by a careless join

You predict on 1 March whether a customer will churn in March. Your training query joins each customer to their current total_support_tickets, computed today in October. Customers who churned had a burst of tickets in March and April, so the feature is strongly predictive offline. In production on 1 March, those future tickets do not exist yet. The fix is a point-in-time correct join: for each training row, use only feature values with a timestamp at or before the prediction time.

-- point-in-time join: latest feature value at or before the label's cut-off
SELECT l.customer_id, l.cutoff_date, f.tickets_30d
FROM labels l
JOIN features f
  ON f.customer_id = l.customer_id
 AND f.as_of = (SELECT MAX(as_of) FROM features
                WHERE customer_id = l.customer_id AND as_of <= l.cutoff_date);

What a feature store does

A feature store is a system for defining features once and serving the same values to training and to production. The main parts:

• Feature definitions: the transformation code, written once and registered with a name, owner and version (for example customer.tickets_30d).
• Offline store: the full history of feature values with timestamps, usually in a warehouse or files, used to build training sets with point-in-time joins.
• Online store: the latest value per entity in a fast key-value database, so an API can fetch features in a few milliseconds.
• Materialisation: jobs that compute features and write them to both stores from the same definition.
• Registry and discovery: a catalogue so another team can reuse tickets_30d instead of rebuilding it slightly differently.

Examples include the open-source Feast and Hopsworks, and managed stores in Databricks, Vertex AI and SageMaker.

           feature definition (one piece of code)
                 /                         \
     offline store (history)        online store (latest)
          |                                |
  point-in-time training set        low-latency lookup in the API

Do you need one?

Not always. A feature store earns its cost when several models share features, when online serving needs fresh precomputed aggregates, or when point-in-time correctness is hard to get right by hand. A single batch model can avoid skew more cheaply by sharing one feature function between the training pipeline and the scoring job, and by logging the features actually used at prediction time so you can compare them with training data.

Catching skew

• Log serving features. Store the exact feature vector each prediction used.
• Compare distributions. Regularly compare logged serving features with the training set (the drift tests in module 3 work here too).
• Replay test. Take a sample of production requests, recompute their features with the training code, and check that the values match within a tolerance.

Common mistakes

• Rewriting feature logic for production instead of reusing the training code.
• Joining on "current" values when building historical training data.
• Ignoring defaults. Training drops rows with missing values; serving fills them with 0. The model never saw zeros there.
• Buying a platform first. Adopting a feature store before you have a feature-sharing problem.

Quick recap

• Training/serving skew: production feature values differ from training ones, so performance drops silently.
• Causes: two implementations, different freshness, and point-in-time leakage.
• Feature stores define features once and serve them to an offline store (history) and an online store (latest).
• Small teams can share one feature function and log serving features instead.

Try it

Open the Do tab and take the quiz to check you can spot skew and leakage in realistic setups.