Hypothesis tests and p-values

Why this matters

Did the new website layout really raise sign-ups, or did we get a lucky week? Is this coin, die or production process behaving differently from what we expect? A hypothesis test is a standard procedure for asking whether data is surprising under a "nothing special is going on" assumption. You will see p-values in A/B tests, clinical studies, lab method validation and quality reports, and you will need to read them without being fooled.

The logic in four steps

1. State two hypotheses.
   • The null hypothesisThe default claim of no effect or no difference (H0) that a test tries to find evidence against. H0: no effect, no difference, the default. "The coin is fair: p = 0.5."
   • The alternative H1: what you suspect. "The coin is not fair: p ≠ 0.5."
2. Choose a significance levelThe threshold alpha, chosen before looking at the data, at or below which a p-value leads you to reject H0. α before looking at the data. α = 0.05 is common.
3. Compute a test statistic that measures how far the data is from what H0 predicts, in standard-error units.
4. Find the p-valueThe probability, assuming H0 is true, of a result at least as extreme as the one observed.: the probability, assuming H0 is true, of getting a result at least as extreme as the one observed. If p ≤ α, reject H0. Otherwise, fail to reject it.

It works like a trial: H0 is "innocent". You only convict when the evidence would be very surprising if the defendant were innocent.

A z-test for a proportion

For a yes/no outcome with n trials, x successes and a null value p0:

p_hat = x / n
SE    = sqrt(p0 * (1 - p0) / n)      (the SE under H0)
z     = (p_hat - p0) / SE
two-sided p-value = 2 * (1 - cdf(|z|))

This uses the normal approximation, which is reasonable when n * p0 and n * (1 − p0) are both at least about 10.

Worked example: is the coin fair?

You flip a coin 100 times and get 60 heads.

p_hat = 60 / 100 = 0.60
SE    = sqrt(0.5 * 0.5 / 100) = 0.05
z     = (0.60 - 0.50) / 0.05 = 2.0
p     = 2 * (1 - cdf(2.0)) = 2 * (1 - 0.9772) = 0.0455

p = 0.046 ≤ 0.05, so at α = 0.05 you reject H0: 60 heads out of 100 would be fairly surprising from a fair coin. With 56 heads, z = 1.2 and p ≈ 0.23: not surprising, so you fail to reject H0.

Worked example: A/B sign-ups

A sign-up page historically converts 10% of visitors. With a new design, 1,000 visitors produce 124 sign-ups.

p_hat = 0.124
SE    = sqrt(0.10 * 0.90 / 1000) ≈ 0.00949
z     = 0.024 / 0.00949 ≈ 2.53
p     ≈ 0.011

Strong evidence the conversion rate is no longer 10%. The size of the change (about 2.4 percentage points) is a separate question from whether it is real.

Reading p-values correctly

A p-value of 0.03 means: if H0 were true, results this extreme would happen about 3% of the time.

It does not mean:

• "There is a 3% chance H0 is true." The p-value assumes H0; it cannot also give its probability.
• "There is a 97% chance the effect is real."
• "The effect is large or important." With huge samples, tiny and useless effects become "significant". Always report the effect size, ideally with a confidence interval.

"Fail to reject H0" is not "H0 is proven". It means the data was not surprising enough; the effect may exist but be too small for this sample to detect.

Two kinds of error

• Type I errorA false positive: rejecting a null hypothesis that is actually true; its rate is alpha. (false positive): rejecting a true H0. Its rate is α.
• Type II errorA false negative: failing to reject a null hypothesis that is actually false. (false negative): failing to reject a false H0. Bigger samples reduce it; this is the power of a test.

Lowering α to 0.01 cuts false positives but makes real effects harder to detect. The right balance depends on the cost of each mistake.

p-hacking

If you test 20 unrelated things at α = 0.05, you expect about one "significant" result by pure chance. Testing many variables, subgroups or stopping points and reporting only the ones that cross 0.05 is called p-hackingRunning many tests or analyses and reporting only those that cross the significance threshold., and it produces findings that do not replicate. Decide on the hypothesis, the test and the sample size before you look.

Common mistakes

• Treating p as the probability that H0 is true.
• Confusing "statistically significant" with "important".
• Saying "we proved H0" after a non-significant result.
• Running many tests and reporting only the winners.

Quick recap

• H0 is the default; the p-value is P(data this extreme | H0).
• Reject H0 when p ≤ α (chosen in advance).
• z-test for a proportion: z = (p̂ − p0) / sqrt(p0(1 − p0)/n).
• Report the effect size, not just the p-value; beware multiple testing.

Try it

In the Do tab, write the z statistic and test the coin and the sign-up page.