Choose between one-sample, paired, Welch and chi-square tests, and compute their test statistics.
Why this matters
Statistics Fundamentals introduced the logic of a hypothesis test: assume no effect (the null hypothesis), measure how surprising the data is, and report a p-value. In practice the hard part is choosing which test. Did a new catalyst change yield? Did a redesign change the sign-up rate? Is defect type related to shift? Each question has a standard test, and picking the wrong one is one of the most common errors reviewers flag.
Step 1: what kind of data?
- A numeric outcome (yield, time, concentration) compared across one or two groups: a t-test.
- Counts in categories (pass/fail by shift, defect type by machine): a chi-square test.
The t-test family
All t-tests share one shape:
t = (observed difference - difference under H0) / standard error
and compare t to a t distribution with some degrees of freedom (df). The versions differ in what the difference and standard error are.
| situation | test | in SciPy |
|---|---|---|
| One group vs a target value (is mean fill 500 ml?) | one-sample t | stats.ttest_1samp(x, 500) |
| Same units measured twice (before/after on each patient or machine) | paired t | stats.ttest_rel(before, after) |
| Two independent groups (catalyst A vs catalyst B batches) | Welch two-sample t | stats.ttest_ind(a, b, equal_var=False) |
A paired test is just a one-sample t-test on the differences. Pairing removes unit-to-unit variation, so when the design is paired, use the paired test: treating paired data as independent throws away power.
For two independent groups, prefer Welch's test, which does not assume equal variances. The older "Student" version (equal_var=True) is only safe when the variances really are similar, and Welch loses almost nothing when they are.
Welch's formulas
SE = sqrt(s1²/n1 + s2²/n2)
t = (x̄1 - x̄2) / SE
df = (s1²/n1 + s2²/n2)² / [ (s1²/n1)²/(n1-1) + (s2²/n2)²/(n2-1) ]
Worked example: two catalysts
Yields (%) from 8 batches with catalyst A and 7 with catalyst B:
A: 12.1 11.4 13.0 12.7 11.9 12.5 13.3 12.2 mean 12.39
B: 11.2 11.8 10.9 11.5 12.0 11.1 11.6 mean 11.44
Welch's test gives t ≈ 3.58 with df ≈ 12.0, and a two-sided p-value of about 0.004. The data is hard to explain if the catalysts were equivalent. Always report the effect alongside the p-value: A's mean yield is about 0.95 percentage points higher.
Assumptions: independent observations, and either roughly normal data in each group or samples large enough for the CLT (module 2). With small, heavily skewed samples, consider a rank test (stats.mannwhitneyu) or a bootstrap.
The chi-square test of independence
For two categorical variables, arrange counts in a contingency table. Under the null hypothesis of independence, the expected count in each cell is
expected = (row total × column total) / grand total
and the statistic adds up how far the observed counts are from expected:
χ² = sum over cells of (observed - expected)² / expected
df = (rows - 1) × (columns - 1)
Worked example: pass rate by shift
| pass | fail | total | |
|---|---|---|---|
| Day shift | 30 | 20 | 50 |
| Night shift | 15 | 35 | 50 |
| total | 45 | 55 | 100 |
Expected "day, pass" = 50 × 45 / 100 = 22.5, and similarly 27.5, 22.5, 27.5 for the other cells. Then
χ² = (30-22.5)²/22.5 + (20-27.5)²/27.5 + (15-22.5)²/22.5 + (35-27.5)²/27.5 ≈ 9.09
df = (2-1)(2-1) = 1 p ≈ 0.0026
Pass rate depends on shift (60% vs 30%). In SciPy: stats.chi2_contingency(table). Note that for 2×2 tables SciPy applies a continuity correction by default, so its statistic (about 7.92 here) is slightly smaller; pass correction=False to match the hand calculation.
Assumption: expected counts should be at least about 5 in each cell. For small tables with small counts, use Fisher's exact test (stats.fisher_exact).
A related test, chi-square goodness of fit (stats.chisquare), compares one categorical variable to expected proportions, such as checking that a die is fair.
A quick decision guide
- Categories and counts? Chi-square (or Fisher for small counts).
- Numbers, one group vs a target? One-sample t.
- Numbers, same units twice? Paired t.
- Numbers, two separate groups? Welch t.
- More than two groups? ANOVA, covered in Regression & Experimental Design.
Common mistakes
- Unpaired test on paired data. You lose the benefit of pairing.
- Running a t-test on proportions or counts in categories. Use chi-square.
- Using percentages in a chi-square table. It needs raw counts.
- Reporting only "p < 0.05". Report the effect size, a confidence interval and the exact p-value.
Recap
- t-tests compare means: one-sample, paired, or Welch for two independent groups.
- Chi-square tests compare observed and expected counts;
df = (r-1)(c-1). - Check assumptions: independence, normality or large n for t; expected counts ≥ 5 for chi-square.
Try it
In the Do tab, compute Welch's t and the chi-square statistic yourself and check them against the worked examples.