Data and concept drift

Why this matters

A model learns patterns from the past and is used on the future. When the world changes, its accuracy decays, usually without any error message. Labels that would reveal the drop often arrive weeks later (did the customer actually churn? was the loan repaid?). Drift detection watches the inputs and, where possible, the relationship between inputs and outcomes, so you notice change early instead of after the damage.

The core idea: two kinds of drift

• Data drift (covariate shift): the distribution of the inputs P(X) changes. A redesigned app makes sessions longer; a marketing campaign brings in younger customers; a new sensor reads 2 degrees higher. The model may still be right for each kind of input, but it now sees inputs it saw rarely in training.
• Concept drift: the relationship between inputs and the target P(y | X) changes. The same transaction pattern that was safe last year is now a common fraud trick; during a recession, the same income predicts a higher default rate. The inputs can look identical while the right answer has changed.

There is also label drift (the share of positives changes) and plain data quality problems (a column suddenly all nulls, units switched from kg to g). Quality checks should run before any statistical drift test: a broken pipeline is not drift.

Measuring data drift for one feature

You compare a reference sample (usually the training data or a known-good period) with a current window (for example the last 7 days of production).

Population Stability Index (PSI)

PSI is widely used in industry, especially in credit risk:

1. Cut the reference data into bins, typically 10 bins at the reference deciles so each holds about 10% of the reference.
2. Compute the share of reference values e_i and current values a_i in each bin.
3. Sum the contributions:

PSI = sum over bins of (a_i - e_i) * ln(a_i / e_i)

A tiny epsilon replaces empty-bin shares of 0 so the logarithm is defined.

Worked example. With two bins, the reference splits 50% / 50% and the current window splits 70% / 30%:

(0.70 - 0.50) * ln(0.70 / 0.50) = 0.20 * 0.3365 = 0.0673
(0.30 - 0.50) * ln(0.30 / 0.50) = -0.20 * -0.5108 = 0.1022
PSI = 0.0673 + 0.1022 = 0.1695

A common rule of thumb reads PSI as: below 0.1 stable, 0.1 to 0.25 moderate shift (investigate), above 0.25 significant shift. These cut-offs are conventions, not laws: tune them for each feature using historical data.

Other tests

• Kolmogorov-Smirnov (KS): the largest gap between two cumulative distributions, for numeric features (scipy.stats.ks_2samp).
• Chi-squared test: for categorical features, compares category counts.
• Wasserstein distance / Jensen-Shannon divergence: smooth distances that do not depend on a p-value.

Be careful with p-values at production volumes: with a million rows, a KS test will call a meaningless 0.1% shift "significant". Effect-size measures such as PSI or a distance with a tuned threshold usually make better alerts.

Detecting concept drift

Concept drift needs outcomes. Options, from best to weakest:

1. Live performance once labels arrive: accuracy, AUC or error on recent predictions, compared with the validation result.
2. Proxy signals that arrive faster: chargebacks for fraud, returns for recommendations, manual review overturn rates.
3. Prediction drift: a shift in the distribution of the model's scores, which can hint at change before labels exist.

Libraries such as Evidently, NannyML and alibi-detect package these checks into reports and dashboards.

Choosing windows

The reference and current windows shape what you detect. A 1-day window is noisy and catches weekday effects; a 90-day window reacts slowly. Many teams compare the last 7 days with the training set and also with the same week last year to separate seasonality from true change.

Common mistakes

• Alerting on every feature. With 200 features, a few will always cross a threshold. Weight by feature importance and alert on the ones that matter.
• Bins from the current data. Bins must come from the reference, or the comparison is meaningless.
• Treating drift as failure. Drift says "look", not "retrain now". Check whether performance actually moved.
• Ignoring seasonality. December is not drift if every December looks like this.

Quick recap

• Data drift: inputs change. Concept drift: the input-outcome relationship changes.
• PSI compares binned shares; under 0.1 stable, 0.1 to 0.25 moderate, over 0.25 significant (tune per feature).
• Run data-quality checks first; use effect sizes rather than raw p-values at scale.
• Confirm concept drift with live labels or proxies before retraining.

Try it

Open the Do tab and implement PSI in NumPy to flag drift in a feature after an app redesign.