Fairness and bias

Why this matters

Models are increasingly used to decide who gets a loan, a job interview, an apartment, extra medical attention or a police visit. A model learns from historical data, and history contains unequal treatment, missing groups and flawed measurements. Without checking, a model can quietly repeat or amplify those problems at scale. Checking a model for group disparities is part of doing the job properly, and in many domains (credit, hiring, housing) it is also a legal obligation.

Where bias comes from

• Historical bias: the labels reflect past decisions. If past hiring favoured one group, a model trained to predict "was hired" learns that preference.
• Representation bias: some groups are under-represented in the training data, so the model is less accurate for them. Face-analysis systems trained mostly on lighter-skinned faces made far more errors on darker-skinned faces.
• Measurement bias: the label is a poor proxy for what you care about. Using past health-care spending as a proxy for health need underestimates need for groups that historically had less access to care.
• Aggregation bias: one model for everyone when relationships differ between groups.
• Deployment bias: a model used for a purpose or population it was not built for.

Removing the sensitive column is not enough

Dropping race or gender from the features ("fairness through unawareness") rarely works. Other features act as proxies: ZIP code, school, first name, shopping patterns and gaps in employment history can all correlate strongly with protected attributes. You usually need the sensitive attribute for auditing, even if the model never uses it.

Measuring disparities

Compute the usual metrics separately for each group and compare. The most common group fairness criteria:

• Demographic parity: each group gets positive decisions at a similar rate (similar selection rates).
• Equal opportunity: among people who truly qualify, each group is approved at a similar rate (similar true positive rates, that is, recall).
• Equalised odds: similar true positive rates and similar false positive rates.
• Calibration: among people given a 70% score, about 70% truly have the outcome, in every group.

Worked example: a loan model

                      group A    group B
applicants              1,000        500
approved                  400        120
selection rate           0.40       0.24
truly creditworthy        600        250
approved & creditworthy   380        110
true positive rate       0.63       0.44

The disparate impact ratio is 0.24 / 0.40 = 0.60. In US employment law, the EEOC's "four-fifths rule" treats a ratio below 0.8 as a sign of possible adverse impact that needs investigating. The equal opportunity gap is 0.63 - 0.44 = 0.19: a creditworthy applicant in group B is much less likely to be approved than one in group A. Both numbers say this model needs a closer look before anyone relies on it.

You cannot satisfy every definition at once

When groups have different base rates, it is mathematically impossible (except in trivial cases) to have calibration, equal false positive rates and equal false negative rates all at the same time. Choosing which criterion matters is a value judgement about which errors harm people most, and it belongs with domain experts, affected communities, legal and compliance staff, not with the data scientist alone. Your job is to measure clearly, explain the trade-offs and document the decision.

What you can do

1. Audit: report accuracy, selection rate, TPR and FPR per group (and for intersections such as age × gender) on held-out data.
2. Fix the data: collect more data for under-represented groups, fix label definitions, remove proxy features that have no legitimate reason to be there.
3. Adjust the model or thresholds: fairness-constrained training, or group-aware thresholds where the law allows it (in some domains it does not; check).
4. Keep a human in the loop for high-stakes decisions and provide a way to appeal.
5. Document: a model card states the intended use, the data, performance per group and known limitations.
6. Monitor after deployment, because populations and behaviour drift.

Open-source tools help with the arithmetic: Fairlearn (MetricFrame computes any metric by group) and IBM's AI Fairness 360. Frameworks such as the NIST AI Risk Management Framework (https://www.nist.gov/itl/ai-risk-management-framework) describe how organisations govern these risks. Rules differ by country and sector and keep changing (for example the EU AI Act), so confirm current requirements with your legal team. Checked on 2026-10-23.

Common mistakes

• Reporting one overall accuracy number. A model can be 95% accurate overall and much worse for a small group.
• Assuming dropping the sensitive column makes the model fair.
• Treating fairness as a one-off check instead of something monitored.
• Choosing a fairness metric to make the numbers look good rather than the one that matches the harm.
• Ignoring small groups because their sample sizes are noisy; report uncertainty instead of leaving them out.

Quick recap

• Bias enters through history, representation, measurement, aggregation and deployment.
• Proxies mean that removing a sensitive column does not remove its influence.
• Compare selection rates, TPR and FPR across groups; the four-fifths ratio and TPR gaps are simple flags.
• Fairness definitions conflict; choose with stakeholders, document with a model card, keep monitoring.

Try it

Take the quiz on the Do tab to practise computing and interpreting group disparities.