Ethics of models

Why this matters

Models decide who gets a loan, what an insurance policy costs, which patients get extra care and where police patrol. A model can be mathematically correct and still treat people unfairly, mislead decision-makers or be used for something it was never designed to do. Statisticians, actuaries and operations research analysts are often the people best placed to notice. Professional codes expect them to speak up.

Where bias comes from

Biased data. If the historical data reflects past discrimination, a model trained on it learns that pattern. A well-known example: in 2018, Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it penalised CVs that mentioned women's organisations, because it had learned from a decade of mostly male hires.

Unrepresentative samples. A survey or training set that under-represents some groups produces estimates that are less accurate for them.

Proxy variables. A model may not use a protected characteristic such as race directly, but other variables (postcode, school, shopping patterns) can stand in for it. This is called proxy discriminationUnfair treatment that arises when variables like postcode stand in for a protected characteristic the model does not use directly..

Wrong target. A model can optimise the wrong thing. A widely cited 2019 study in Science by Obermeyer and colleagues found that a health-care algorithm used past health spending as a stand-in for health need. Because less money had historically been spent on Black patients with the same level of illness, the algorithm underestimated their need for extra care.

Feedback loopsCycles where a model's decisions change the data it later learns from, reinforcing its own pattern regardless of reality.. If a model sends more inspections to one area, more problems are found there, which makes the model send even more inspections, regardless of where problems really are.

Different ideas of fairness

There is no single mathematical definition of fairness. Common ones include:

• Equal accuracy: the model is equally accurate for each group.
• Equal error rates: for example, the same false positive rate across groups.
• CalibrationA fairness criterion requiring that a predicted risk, such as 30%, means the same actual rate for every group.: a predicted 30% risk means 30% for every group.

In 2016 ProPublica analysed a criminal risk-scoring tool (COMPAS) and reported that Black defendants who did not reoffend were more likely to be wrongly labelled high risk. The tool's maker argued it was equally calibrated across groups. Researchers later showed that, when groups have different underlying rates, it is mathematically impossible to satisfy all these fairness criteria at once. Choosing between them is an ethical and legal decision, not only a technical one, and it should be made openly.

Misuse beyond bias

• Using a model outside its purpose or range. A model built to rank applicants for a scholarship is not valid for deciding who gets a job.
• Hiding uncertainty so a decision looks more certain than it is (lesson 1 of this module).
• Data dredging (p-hackingTesting many variations until one gives a significant result and reporting only that one; also called data dredging.): testing many variations until one gives a "significant" result, then reporting only that one.
• Goodhart's lawThe idea that when a measure becomes a target, people change behaviour to hit it and it stops measuring what it used to.: when a measure becomes a target, people change their behaviour to hit the number, and it stops measuring what it used to.
• Pressure to deliver a number. Being asked to adjust assumptions until a project looks profitable or a reserve looks smaller.

Worked example: reviewing an insurance pricing model

An actuary reviews a new car insurance model that uses 40 variables. Questions to ask:

1. Legal: which characteristics are prohibited or restricted by insurance regulation in this jurisdiction? Is any variable a close proxy for one?
2. Data: does the training data under-represent any group or region?
3. Outcomes: compare prices and error rates across groups and regions. Are differences explained by genuine differences in risk?
4. Explainability: can the insurer explain to a customer and a regulator why their price is what it is?
5. Monitoring: who checks the model after launch, and how often?

If something looks wrong, the actuary documents it, raises it with the responsible people and, if needed, follows the professional code's guidance on escalation.

Professional codes

Each profession has written standards. The American Statistical Association publishes Ethical Guidelines for Statistical Practice at amstat.org. Actuaries follow their society's Code of Professional Conduct (see soa.org and casact.org) and, in the US, the Actuarial Standards of Practice issued by the Actuarial Standards Board. INFORMS publishes ethics guidelines for operations research and analytics at informs.org. Checked on 2026-10-23.

Common themes: be honest about methods and limitations, respect privacy, avoid conflicts of interest, do not let clients or employers pressure you into misleading results, and consider who is affected by your work.

Common mistakes

• "The model is just math, so it is neutral." Data, targets and choices carry values.
• Removing a protected variable and assuming the problem is solved. Check for proxies.
• Checking only overall accuracy. Look at results for each group.
• Staying silent. Professional codes expect you to raise concerns.

Quick recap

• Bias enters through data, samples, proxies, wrong targets and feedback loops.
• Fairness has several definitions that cannot all hold at once; choose openly.
• Watch for misuse: wrong purpose, hidden uncertainty, p-hacking, Goodhart's law, pressure.
• Know your profession's code: ASA guidelines, actuarial codes and ASOPs, INFORMS ethics guidelines.

Try it

This lesson has no activity. Watch the videos, then take the quiz in the Do tab.