Anomaly detection

Why this matters

Fraudulent payments, failing machines, hacked accounts and mislabelled lab samples have something in common: they are rare and different. You usually have few or no labelled examples of them, so supervised learning is hard. Anomaly detection scores how unusual each row is so that people can review the strangest ones first. Done well, it turns a million transactions into a short list worth looking at.

Kinds of anomalies

• Point anomalies: a single row is extreme, such as a $40,000 purchase on a card that usually spends $50.
• Contextual anomalies: normal in one context, odd in another, such as 30 °C in January.
• Collective anomalies: a group of rows that is odd together, such as a burst of logins from one IP address.

This lesson focuses on point anomalies in tabular data.

Simple baselines first

For one numeric column, classic rules work well:

• z-score: flag |x - mean| / sd > 3.
• IQR rule: flag values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR (the box-plot rule, robust to the outliers themselves).

They break down with several features. A sensor reading can have a normal temperature and a normal vibration level individually, yet the combination is impossible for a healthy machine. You need a multivariate method.

The isolation forest idea

Instead of modelling what normal looks like, an isolation forest asks: how easy is this point to isolate?

Build a tree by repeatedly picking a random feature and a random threshold between that feature's minimum and maximum, splitting the data until each point sits alone. A normal point lives in a dense crowd and needs many random cuts to separate. An outlier sits far from the crowd, so a random cut isolates it in just a few steps.

dense normal point:  ~8-10 random splits to isolate
far-away outlier:    ~2-3 random splits to isolate

One random tree is noisy, so the forest builds many (often 100), each on a small random subsample of rows (256 by default), and averages each point's path length (the depth at which it gets isolated). Small subsamples actually help: outliers are less likely to be masked by other outliers.

The anomaly score

Path lengths are normalised by c(n), the average path length of an unsuccessful search in a binary search tree of n points:

c(n) = 2 * (ln(n - 1) + 0.5772) - 2 * (n - 1) / n       (c(2) = 1, c(1) = 0)
score(x) = 2 ^ ( -mean_path(x) / c(sample_size) )

(0.5772 is the Euler–Mascheroni constant.) The score is between 0 and 1:

• close to 1: isolated very quickly, a strong anomaly;
• around 0.5: path length is average, nothing special;
• well below 0.5: deep inside the crowd, clearly normal.

c is also used inside the trees: when the depth limit stops a branch that still holds m points, the path gets c(m) added as an estimate of the remaining depth.

Worked example

With a subsample of 256, c(256) = 2 * (ln 255 + 0.5772) - 2 * 255/256 ≈ 10.24. A point isolated after 3 splits on average scores 2^(-3/10.24) ≈ 0.82, a strong anomaly. A point that needs 10.24 splits scores exactly 0.5.

Turning scores into flags

Scores rank rows; you still need a cut-off. In practice you pick a review budget: "the analysts can check 2% of readings a day", so flag the top 2%. scikit-learn calls this contamination. If you do have a few confirmed anomalies, use them to check precision among the flagged rows (Lesson 6's metrics).

from sklearn.ensemble import IsolationForest

iso = IsolationForest(n_estimators=200, contamination=0.02, random_state=0).fit(X_train)
flags = iso.predict(X_new) == -1        # -1 = anomaly
strength = -iso.score_samples(X_new)    # higher = more anomalous

Isolation forests need no feature scaling (cuts are random within each feature's range) and are fast. Alternatives include Local Outlier Factor (compares local density), one-class SVM and, for complex data, autoencoders.

Common mistakes

• Treating every flag as a confirmed problem. An anomaly is unusual, not necessarily bad; a human or a follow-up rule decides.
• Training on data full of the anomalies you want to find, which makes them look normal. Train on a period you believe is mostly clean when you can.
• Including ID or timestamp columns, which makes every new row look unusual.
• Ignoring context, such as comparing night-time and daytime readings together.
• Setting the contamination rate without thinking about review capacity.

Quick recap

• Anomaly detection ranks rows by how unusual they are, usually without labels.
• z-scores and IQR rules work for single columns; multivariate data needs methods like isolation forests.
• Isolation forests isolate points with random splits; short average paths mean anomalies.
• Score = 2^(-mean_path / c(n)); near 1 is anomalous, about 0.5 is ordinary. Flag the top share your team can review.

Try it

On the Do tab, write the normalising constant, the score and the flagging rule, then find the faulty sensor readings.