Spot Simpson's paradox, misleading charts and other ways true numbers tell false stories.
Why this matters
Most misleading statistics are not lies. Every number can be correct and the conclusion can still be wrong, because of how the data was grouped, charted or described. As a data analyst, scientist or engineer you will both read and produce these numbers. Being able to say "that chart's axis starts at 95" or "that comparison ignores group size" in a meeting is one of the most valuable skills this course teaches.
Simpson's paradox
A trend that appears in every group can reverse when the groups are combined. The classic shape: two treatments for kidney stones.
Treatment A Treatment B
small stones 81/87 = 93% 234/270 = 87%
large stones 192/263 = 73% 55/80 = 69%
all patients 273/350 = 78% 289/350 = 83%
A has the higher success rate for small stones and for large stones, yet B looks better overall. How? Doctors gave A mostly to the harder large-stone cases (263 of A's 350) and B mostly to easy small-stone cases (270 of B's 350). Stone size is a confounder: it affects both which treatment was given and the chance of success. The combined numbers mostly compare case mix, not treatments.
The lesson: when groups differ in size or composition, look at the breakdown, and ask what variable might drive both the "treatment" and the outcome. The same thing happens with admission rates across departments, hospital death rates (good hospitals take the sickest patients), and A/B tests where traffic sources shift during the test.
Misleading charts
Truncated axis. A bar chart of conversion rates of 96.1% vs 96.8% with the y-axis starting at 95% makes a 0.7-point difference look like a doubling. Bar charts should start at zero, because bar length is the message. Line charts may zoom in to show change, but the axis should be labelled clearly.
Inconsistent scales. Two charts side by side with different y-axis ranges invite a false comparison. So does an x-axis with uneven time steps (2015, 2020, 2021, 2022).
Area and 3-D tricks. Doubling both the width and height of an icon to show "twice as much" makes it look four times as big. 3-D pie charts distort the slices nearest the viewer.
Cherry-picked windows. Starting a stock chart at a low point or ending a temperature chart at an unusual year can show almost any trend you like. Ask why the chart starts and ends where it does.
Dual axes. Two lines on different y-axes can be made to cross or move together by rescaling either axis.
Misleading numbers
Relative vs absolute risk. "Doubles the risk" sounds alarming. If the risk goes from 1 in 10,000 to 2 in 10,000, the absolute increase is 0.01 percentage points. Report both.
Percent of what? "Sales up 50%!" from 2 units to 3 means little. A small base makes percent changes huge.
Averages hiding spread. "Average response time is 200 ms" can hide 5% of requests taking 4 seconds. Report medians, percentiles (p95, p99) and spread.
Survivorship and selection. "Our graduates earn $90,000" may count only graduates who answered the survey or who completed the program.
Correlation presented as cause. "People who drink coffee live longer" from observational data does not show that coffee extends life. Look for confounders such as income or overall health.
Statistical vs practical significance. With a million users, a 0.1% improvement can have a tiny p-value and still not be worth building.
Regression to the mean. Extreme results tend to be followed by more ordinary ones. A team that hires a new coach after its worst season will probably improve next season regardless. Without a comparison group, you cannot credit the coach.
A checklist for any claim
- Who collected the data, and how was the sample chosen?
- Compared with what? Is there a baseline or control group?
- Absolute numbers or only percentages? What is the base?
- Could a third variable explain it? Look at the breakdown by group.
- Does the chart's axis start at zero (for bars)? Are scales consistent?
- Is the difference large enough to matter, not just "significant"?
- Why does the time window start and end where it does?
Quick recap
- Simpson's paradox: a confounder plus unequal group mix can reverse a trend when groups are combined.
- Charts mislead through truncated axes, inconsistent scales, area effects, cherry-picked windows and dual axes.
- Numbers mislead through relative-only risk, small bases, averages without spread, selection, causal claims from correlation and regression to the mean.
- Ask: compared with what, out of how many, and what else could explain it?
Try it
Open the Do tab and test the checklist on the quiz scenarios.