Hypothesis tests and p-values

Why this matters

Did the new website layout really raise sign-ups, or did we get a lucky week? Is this coin, die or production process behaving differently from what we expect? A hypothesis test is a standard procedure for asking whether data is surprising under a "nothing special is going on" assumption. You will see p-values in A/B tests, clinical studies, lab method validation and quality reports, and you will need to read them without being fooled.

The logic in four steps

1. State two hypotheses.
   • The null hypothesis H0: no effect, no difference, the default. "The coin is fair: p = 0.5."
   • The alternative H1: what you suspect. "The coin is not fair: p ≠ 0.5."
2. Choose a significance level α before looking at the data. α = 0.05 is common.
3. Compute a test statistic that measures how far the data is from what H0 predicts, in standard-error units.
4. Find the p-value: the probability, assuming H0 is true, of getting a result at least as extreme as the one observed. If p ≤ α, reject H0. Otherwise, fail to reject it.

It works like a trial: H0 is "innocent". You only convict when the evidence would be very surprising if the defendant were innocent.

A z-test for a proportion

For a yes/no outcome with n trials, x successes and a null value p0:

p_hat = x / n
SE    = sqrt(p0 * (1 - p0) / n)      (the SE under H0)
z     = (p_hat - p0) / SE
two-sided p-value = 2 * (1 - cdf(|z|))

This uses the normal approximation, which is reasonable when n * p0 and n * (1 − p0) are both at least about 10.

Worked example: is the coin fair?

You flip a coin 100 times and get 60 heads.

p_hat = 60 / 100 = 0.60
SE    = sqrt(0.5 * 0.5 / 100) = 0.05
z     = (0.60 - 0.50) / 0.05 = 2.0
p     = 2 * (1 - cdf(2.0)) = 2 * (1 - 0.9772) = 0.0455

p = 0.046 ≤ 0.05, so at α = 0.05 you reject H0: 60 heads out of 100 would be fairly surprising from a fair coin. With 56 heads, z = 1.2 and p ≈ 0.23: not surprising, so you fail to reject H0.

Worked example: A/B sign-ups

A sign-up page historically converts 10% of visitors. With a new design, 1,000 visitors produce 124 sign-ups.

p_hat = 0.124
SE    = sqrt(0.10 * 0.90 / 1000) ≈ 0.00949
z     = 0.024 / 0.00949 ≈ 2.53
p     ≈ 0.011

Strong evidence the conversion rate is no longer 10%. The size of the change (about 2.4 percentage points) is a separate question from whether it is real.

Reading p-values correctly

A p-value of 0.03 means: if H0 were true, results this extreme would happen about 3% of the time.

It does not mean:

• "There is a 3% chance H0 is true." The p-value assumes H0; it cannot also give its probability.
• "There is a 97% chance the effect is real."
• "The effect is large or important." With huge samples, tiny and useless effects become "significant". Always report the effect size, ideally with a confidence interval.

"Fail to reject H0" is not "H0 is proven". It means the data was not surprising enough; the effect may exist but be too small for this sample to detect.

Two kinds of error

• Type I error (false positive): rejecting a true H0. Its rate is α.
• Type II error (false negative): failing to reject a false H0. Bigger samples reduce it; this is the power of a test.

Lowering α to 0.01 cuts false positives but makes real effects harder to detect. The right balance depends on the cost of each mistake.

p-hacking

If you test 20 unrelated things at α = 0.05, you expect about one "significant" result by pure chance. Testing many variables, subgroups or stopping points and reporting only the ones that cross 0.05 is called p-hacking, and it produces findings that do not replicate. Decide on the hypothesis, the test and the sample size before you look.

Common mistakes

• Treating p as the probability that H0 is true.
• Confusing "statistically significant" with "important".
• Saying "we proved H0" after a non-significant result.
• Running many tests and reporting only the winners.

Quick recap

• H0 is the default; the p-value is P(data this extreme | H0).
• Reject H0 when p ≤ α (chosen in advance).
• z-test for a proportion: z = (p̂ − p0) / sqrt(p0(1 − p0)/n).
• Report the effect size, not just the p-value; beware multiple testing.

Try it

In the Do tab, write the z statistic and test the coin and the sign-up page.