The bootstrap

Why this matters

In Statistics Fundamentals you built a confidence interval for a mean with a formula. Formulas exist for a few statistics (means, proportions, regression slopes) under some assumptions. But what is the margin of error on a median repair time, a 90th percentile response time, a ratio of two means, or a model's accuracy? The bootstrap gives a confidence interval for almost any statistic, using only your data and a computer. It is one of the most practical ideas in modern statistics.

The core idea

If you could, you would draw many fresh samples from the population, compute the statistic on each, and look at how much it varies. You cannot, because you have one sample. The bootstrap's trick is to treat your sample as a stand-in for the population and draw new samples from it:

1. Take your data of size n.
2. Draw a resample of size n from it with replacement (some values appear twice, some not at all).
3. Compute the statistic on the resample.
4. Repeat B times (typically 2,000 to 10,000).
5. The spread of the B results estimates the sampling variability. For a 95% percentile interval, take the 2.5th and 97.5th percentiles of them.

"With replacement" is essential. Without it, every resample would be the original data shuffled, and the statistic would never change.

Worked example: median repair time

A maintenance team logged 12 repair times (hours). The data is skewed by two long jobs, so the median is the better summary.

import numpy as np

times = np.array([1.2, 0.8, 2.5, 1.1, 0.9, 6.0, 1.4, 1.7, 0.7, 2.1, 9.5, 1.3])
print(np.median(times))                    # 1.35

rng = np.random.default_rng(7)
boot = []
for _ in range(5000):
    resample = rng.choice(times, size=len(times), replace=True)
    boot.append(np.median(resample))

lo, hi = np.percentile(boot, [2.5, 97.5])
print(lo, hi)                              # roughly 1.0 to 2.3

Report it as: "median repair time 1.35 hours (95% bootstrap CI about 1.0 to 2.3 hours)". The interval is not symmetric around 1.35, and that is a feature: it reflects the skew in the data, which a plus-or-minus formula would hide.

Bootstrapping other statistics

Swap np.median for any function of the data:

• np.mean, to check the textbook interval.
• lambda x: np.percentile(x, 90) for a 90th-percentile latency.
• A difference between two groups: resample each group separately, then subtract.
• A model's accuracy on a test set: resample the test rows.

scipy.stats.bootstrap(...) implements this (with a better "BCa" interval by default), and you will see it in data-science code. Writing the loop yourself once makes it clear what the library does.

When the bootstrap works, and when it does not

The bootstrap assumes your sample represents the population and that observations are independent. It works well for means, medians, percentiles in the middle of the data, and most smooth statistics, given a reasonable n (say 20 or more).

It struggles with:

• Very small samples. Twelve values give a rough interval; five values give a misleading one.
• Extremes. The maximum of a resample can never exceed the sample maximum, so bootstrapping the max or the 99.9th percentile underestimates uncertainty.
• Dependent data. Time series and clustered data (many readings per patient, per machine) need block or cluster resampling, or the interval will be too narrow.

How many resamples?

B controls only the Monte Carlo error of the interval endpoints, not the underlying uncertainty from your sample size. 1,000 resamples are fine for a quick look; use 5,000 to 10,000 for anything you publish. Always seed the generator so the interval is reproducible.

Common mistakes

• Resampling without replacement. Every resample becomes identical to the data.
• Resampling a different size. Each resample must have the same n as the original.
• Thinking more resamples means more certainty. Only more real data narrows the interval.
• Ignoring structure. Resample whole clusters (patients, batches) when observations come in groups.

Recap

• Resample your data with replacement, recompute the statistic, repeat thousands of times.
• The 2.5th and 97.5th percentiles of the results form a 95% percentile interval.
• Works for medians, percentiles, ratios and model metrics, not just means.
• Needs independent data and a reasonable sample; struggles with extremes.

Try it

In the Do tab, write a reusable bootstrap function and use it for a median and a mean.