Use the confusion matrix, precision, recall, F1 and ROC AUC to judge classifiers when classes are imbalanced.
Why this matters
A model is only as good as the number you use to judge it. Choose the wrong metric and you can ship a model that looks excellent and does nothing useful. This is most dangerous with imbalanced classes, which are the norm in practice: fraud, disease, equipment failure and churn are all rare. A metric must match what the model is for and what its mistakes cost.
The accuracy trap
A disease affects 2% of 1,000 patients (20 people). A "model" that always says "healthy" is right 980 times: 98% accuracy, and it never finds a single sick patient. Accuracy counts all correct answers equally, so the majority class dominates it.
The confusion matrix
For a binary classifier, every prediction lands in one of four cells:
predicted positive predicted negative
actual positive TP FN
actual negative FP TN
- TP (true positive): sick, flagged. FN (false negative): sick, missed.
- FP (false positive): healthy, flagged. TN (true negative): healthy, cleared.
Precision, recall and F1
precision = TP / (TP + FP) of the cases we flagged, how many were real?
recall = TP / (TP + FN) of the real cases, how many did we catch?
F1 = 2 * precision * recall / (precision + recall)
Recall is also called sensitivity or the true positive rate. F1 is the harmonic mean, which is high only when both are high.
Worked example
A screening model on the 1,000 patients flags 50 people, 16 of whom are sick:
TP = 16, FP = 34, FN = 4, TN = 946
precision = 16 / 50 = 0.32
recall = 16 / 20 = 0.80
F1 = 2 * 0.32 * 0.80 / 1.12 ≈ 0.457
accuracy = (16 + 946) / 1000 = 0.962
Accuracy went down compared with the useless "always healthy" model (0.962 vs 0.98), yet this model catches 80% of sick patients. For a cheap follow-up test, that trade is excellent. Precision and recall show it; accuracy hides it.
When the denominator is zero (the model flags nobody), precision is undefined. Libraries report 0 and warn; your own code should handle it too.
Thresholds and the trade-off
A classifier outputs scores or probabilities, and the threshold decides who is flagged. Lowering the threshold flags more rows: recall rises, precision usually falls. There is no free lunch; you choose a point based on cost:
- Recall first: cancer screening, fraud review queues, safety alerts, where a miss is expensive.
- Precision first: spam filtering that deletes mail, automated account bans, where false alarms are expensive.
A practical recipe: "pick the threshold with the highest precision among those reaching at least 90% recall", chosen on the validation set.
ROC and AUC
The ROC curve plots the true positive rate (recall) against the false positive rate FP / (FP + TN) as the threshold sweeps from high to low. The area under it (AUC) summarises ranking quality in one threshold-free number:
- AUC = 1.0: every positive scores above every negative.
- AUC = 0.5: no better than random ordering.
AUC has a useful interpretation: it is the probability that a randomly chosen positive scores higher than a randomly chosen negative (ties count half). That is exactly how you will compute it from scratch:
import numpy as np
pos, neg = scores[y == 1], scores[y == 0]
greater = (pos[:, None] > neg[None, :]).mean()
ties = (pos[:, None] == neg[None, :]).mean()
auc = greater + 0.5 * ties
With heavy imbalance, ROC AUC can look comfortable while precision is poor, because the false positive rate divides by the huge negative count. The precision-recall curve and its average precision (average_precision_score) are more informative when positives are rare.
In scikit-learn
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
f1_score, roc_auc_score, average_precision_score,
classification_report)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(roc_auc_score(y_test, proba), average_precision_score(y_test, proba))
For regression, the common metrics are MAE (mean absolute error, in label units, robust to outliers), RMSE (punishes big errors more) and R-squared.
Common mistakes
- Reporting accuracy alone on imbalanced data. Always show the baseline and the confusion matrix.
- Choosing the threshold on the test set. Choose it on validation data.
- Computing AUC from hard 0/1 predictions instead of scores, which throws away the ranking.
- Averaging metrics carelessly across classes: macro averaging treats classes equally, micro averaging weights by count.
Quick recap
- Accuracy misleads when classes are imbalanced.
- Precision = how trustworthy the flags are; recall = how many real cases are caught; F1 balances them.
- Thresholds trade precision for recall; set them by cost on validation data.
- ROC AUC measures ranking; prefer precision-recall metrics when positives are rare.
Try it
On the Do tab, compute the confusion matrix, precision, recall, F1 and ROC AUC from scratch, and see why accuracy fools you.