Power and multiple testing

Why this matters

Two errors ruin studies. The first is running a study too small to detect the effect you care about, then concluding "no effect". The second is running so many tests that some come out "significant" by luck. A/B testing teams, lab scientists screening thousands of genes, and analysts slicing a dashboard by every segment meet both problems every week. This lesson gives you the tools to plan against the first and correct for the second.

Errors and power

Power = 1 - β is the probability that your study detects an effect of a given size if it is really there. A common target is 80%.

Power goes up when:

• the true effect δ is larger,
• the noise σ is smaller (better instruments, paired designs),
• the sample size n is larger,
• α is larger (a looser threshold, which costs more false positives).

Computing power for two groups

For comparing two means with n per group, the normal approximation is

SE    = σ × sqrt(2 / n)
power ≈ Φ( |δ| / SE - z(1 - α/2) )

where Φ is the standard normal CDF and z(1 - α/2) is 1.96 for α = 0.05. (The tiny chance of rejecting in the wrong direction is ignored.)

Worked example. A new process might raise mean strength by δ = 5 MPa; the SD between specimens is σ = 10 MPa.

n = 20 per group:  SE = 10 × sqrt(2/20) = 3.16   power ≈ Φ(5/3.16 - 1.96) = Φ(-0.38) ≈ 0.35
n = 63 per group:  SE = 1.78                       power ≈ Φ(2.81 - 1.96) = Φ(0.85) ≈ 0.80

With 20 specimens per group you would miss a real 5 MPa improvement about two times in three. A non-significant result from that study says almost nothing.

Solving for n

Rearranging gives the sample size per group for a target power:

n = 2 × (z(1 - α/2) + z(power))² × σ² / δ²
  = 2 × (1.96 + 0.84)² × 100 / 25 ≈ 62.8  -> 63 per group

from statistics import NormalDist
Z = NormalDist()
n = 2 * (Z.inv_cdf(0.975) + Z.inv_cdf(0.80)) ** 2 * 10**2 / 5**2
print(n)     # 62.79

Libraries such as statsmodels.stats.power.TTestIndPower do this with the exact t distribution and give a slightly larger n. Always round up, and plan for dropouts or failed runs.

The hard part is choosing δ: use the smallest effect that would matter to the business or science, not the effect you hope for.

Multiple testing

Run one test at α = 0.05 with no real effect and you have a 5% chance of a false positive. Run 20 independent tests and

P(at least one false positive) = 1 - 0.95^20 ≈ 0.64

Testing 20 dashboard segments, 20 outcome measures or 20,000 genes guarantees "discoveries" unless you correct.

Bonferroni: control the family-wise error rate

Test each of m hypotheses at α / m. With 10 tests at overall α = 0.05, each needs p ≤ 0.005. This keeps the chance of any false positive at or below 5%. It is simple and safe, but very strict when m is large.

Benjamini-Hochberg: control the false discovery rate

When you screen many hypotheses and can tolerate a few false leads, control the false discovery rate (FDR): the expected share of your discoveries that are false.

1. Sort the m p-values from smallest to largest: p(1) ≤ p(2) ≤ ... ≤ p(m).
2. Find the largest rank k with p(k) ≤ (k / m) × α.
3. Reject the hypotheses with the k smallest p-values (all of them, even if an earlier one missed its own threshold).

Worked example, m = 10, α = 0.05:

sorted p:   0.001  0.008  0.012  0.018  0.040  0.20 ...
threshold:  0.005  0.010  0.015  0.020  0.025  0.03 ...
passes?     yes    yes    yes    yes    no     no

The largest passing rank is 4, so BH rejects 4 hypotheses. Bonferroni (threshold 0.005 for all) rejects only 1. In Python: statsmodels.stats.multitest.multipletests(p, method="fdr_bh"), or SciPy's scipy.stats.false_discovery_control.

Good practice

• Pre-register the primary outcome and the analysis before collecting data.
• Don't peek and stop. Checking an A/B test daily and stopping at the first p < 0.05 inflates false positives; use a fixed sample size or a sequential method designed for it.
• Report every test you ran, not just the significant ones.

Common mistakes

• "Not significant" read as "no effect" from an underpowered study.
• Power calculated after the fact from the observed effect: it adds nothing beyond the p-value.
• Correcting only the tests that looked interesting. The correction must cover every test in the family.

Recap

• Power = chance of detecting a real effect; plan for about 80% at the smallest effect that matters.
• For two means: power ≈ Φ(|δ| / (σ sqrt(2/n)) - 1.96); n ≈ 2(1.96 + 0.84)² σ²/δ².
• Many tests inflate false positives: Bonferroni controls any error, BH controls the false discovery rate.

Try it

In the Do tab, write a power function and the Benjamini-Hochberg procedure.