Histograms and distributions

Why this matters

A mean and a standard deviation compress a whole column into two numbers. That is useful, but it can hide a lot: two peaks, a long tail, a cluster of impossible values from a broken sensor. A histogramA chart of a numerical variable showing how many values fall in each equal-width bin, with touching bars. shows the whole distribution, meaning how often each range of values occurs. Looking at it before you summarise is one of the habits that separates careful analysts from careless ones.

Building a histogram

1. Choose bins: equal-width ranges that cover the data, for example 0 to 10, 10 to 20, 20 to 30 minutes.
2. Count how many values fall in each bin.
3. Draw a bar for each bin, with height equal to the count. The bars touch, because the bins are continuous.

A text version of wait times (minutes) at a help desk:

 0-10  ##########   10
10-20  ############ 12
20-30  ######        6
30-40  ###           3
40-50  #             1
50-60                0
60-70  #             1

Bin width matters. Too few bins hide structure; too many turn the picture into noise. Start with something like 5 to 15 bins and try a couple of widths.

A histogram is not a bar chart. A bar chart shows categories (blood type, material) with gaps between bars, and the order of bars is up to you. A histogram shows a numerical variable on a number line.

Describing shape

Describe a distribution with four things: shape, centre, spread and unusual values.

Symmetric: the left and right halves look roughly alike. Repeated measurements of the same quantity and heights of adults of one sex are classic examples. Mean and median are close.

Right-skewedDescribes a distribution with a long tail to the right, where the mean is greater than the median. (positively skewed): a long tail to the right. Most values are small, a few are large. Wait times, incomes, file sizes, reaction times, city populations. The mean is pulled toward the tail, so mean > median.

Left-skewed (negatively skewed): a long tail to the left. Exam scores on an easy test, age at retirement, time to failure for components that rarely fail early. Mean < median.

BimodalDescribes a distribution with two peaks, often a sign that two groups are mixed together.: two peaks. Often a sign that two groups are mixed together, such as morning and evening shifts, two machines, or two species. Splitting the data by group usually explains it.

The help-desk histogram above is right-skewed: the bulk is under 30 minutes, with a tail out to 60+.

Quartiles and the box plot

QuartilesThe three cut points (Q1, Q2 the median, Q3) that split sorted data into four equal parts. split sorted data into four equal parts. Q1 is the 25th percentile, Q2 the median, Q3 the 75th percentile. The IQR is Q3 − Q1.

A box plotA chart with a box from Q1 to Q3, a line at the median and whiskers to the most extreme non-outliers. draws a box from Q1 to Q3 with a line at the median, and "whiskers" out to the most extreme values that are not outliers. Box plots are compact, which makes them good for comparing several groups side by side.

Flagging outliers: the 1.5 × IQR rule

A widely used rule (Tukey's fencesLimits at Q1 - 1.5 x IQR and Q3 + 1.5 x IQR; values outside them are flagged as possible outliers.):

lower fence = Q1 - 1.5 * IQR
upper fence = Q3 + 1.5 * IQR

Any value outside the fences is flagged as a possible outlier.

Worked example

Wait times: 4, 7, 8, 9, 11, 12, 14, 15, 18, 22, 65. Suppose Q1 = 8 and Q3 = 18 (software may give slightly different quartiles; there are several methods).

IQR = 18 - 8 = 10
lower fence = 8 - 15 = -7
upper fence = 18 + 15 = 33

65 is above 33, so it is flagged. Nothing is below −7.

"Flagged" does not mean "delete". An outlier can be a typing error (6.5 entered as 65), a broken instrument, or a real and important event (the one customer who waited an hour). Investigate first. Remove a value only with a documented reason, and say so in your report.

In Python

import statistics

waits = [4, 7, 8, 9, 11, 12, 14, 15, 18, 22, 65]
q1, q2, q3 = statistics.quantiles(waits, n=4)
print(q1, q2, q3)   # 8.0 12.0 18.0
print(statistics.mean(waits), statistics.median(waits))   # mean 16.8 > median 12: right skew

statistics.quantiles(data, n=4) returns the three cut points between quartiles.

Common mistakes

• Summarising with the mean before looking at the shape.
• Using a histogram for categorical data, or a bar chart with gaps for continuous data.
• Deleting outliers automatically because a rule flagged them.
• Judging skew from one bin width only.

Quick recap

• A histogram shows how often values fall in equal-width bins.
• Describe shape (symmetric, right-skewed, left-skewed, bimodal), centre, spread and unusual values.
• Right skew: mean > median. Left skew: mean < median.
• Fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR flag possible outliers; investigate before removing.

Try it

In the Do tab, finish an outlier detector by writing the two fence formulas.