ANOVA

Why this matters

You have run your randomised experiment: three catalysts, five batches each. Are the catalysts really different, or are the differences in average yield just batch-to-batch noise? Running three separate t-tests (A vs B, A vs C, B vs C) inflates the chance of a false positive. Analysis of variance (ANOVA) answers the overall question with one test, and its table is the standard way results of designed experiments are reported in chemistry, biology, materials and engineering papers.

The core idea

ANOVA compares two estimates of variability:

• Between groups: how far the group means are from the grand mean.
• Within groups: how much individual results scatter around their own group mean (pure noise, if the experiment was done properly).

If all treatment means are truly equal, both estimate the same noise variance and their ratio is near 1. If some treatments differ, the between-group estimate is inflated and the ratio grows. That ratio is the F statistic.

The decomposition

With k groups, n_i observations in group i, N observations in total, group means ȳ_i and grand mean ȳ:

SS_between = sum over groups  n_i * (ȳ_i - ȳ)^2          df = k - 1
SS_within  = sum over all obs (y_ij - ȳ_i)^2              df = N - k
SS_total   = SS_between + SS_within                       df = N - 1

MS = SS / df          F = MS_between / MS_within

Under the null hypothesis (all means equal) and the usual assumptions, F follows an F distribution with (k − 1, N − k) degrees of freedom.

Worked example: three catalysts

Catalyst A: 82 85 88 84 86   mean 85
Catalyst B: 90 92 89 93 91   mean 91
Catalyst C: 84 83 86 85 82   mean 84     grand mean 86.667

Between groups: 5 × [(85 − 86.667)² + (91 − 86.667)² + (84 − 86.667)²] = 5 × 28.667 = 143.33.

Within groups: A deviations −3, 0, 3, −1, 1 give 20; B gives 10; C gives 10; total 40.

Source     SS       df    MS       F
Catalyst   143.33    2    71.67   21.5
Error       40.00   12     3.33
Total      183.33   14

The 5% critical value of F(2, 12) is 3.89, and 21.5 is far above it (p ≈ 0.0001). We reject the hypothesis that all three catalysts give the same mean yield. A useful effect size is η² = SS_between / SS_total = 0.78: catalyst explains 78% of the variation in these runs.

The same thing in Python

from scipy import stats
import statsmodels.formula.api as smf
from statsmodels.stats.anova import anova_lm

a = [82, 85, 88, 84, 86]; b = [90, 92, 89, 93, 91]; c = [84, 83, 86, 85, 82]
print(stats.f_oneway(a, b, c))          # F = 21.5, p ≈ 1e-4

# The regression view: ANOVA is a linear model with a categorical predictor
model = smf.ols("yield_pct ~ C(catalyst)", data=df).fit()
print(anova_lm(model, typ=2))

The second form matters. ANOVA is multiple regression with dummy variables, so everything from module 1 applies: residual diagnostics, adding a block term (+ C(day)), and adding covariates (ANCOVA).

After a significant F: which groups differ?

The F test says "not all equal". To say which pairs differ, use a multiple-comparison procedure that controls the family-wise error rate:

from statsmodels.stats.multicomp import pairwise_tukeyhsd
print(pairwise_tukeyhsd(df["yield_pct"], df["catalyst"], alpha=0.05))

Tukey's HSD compares all pairs. Here it finds B higher than both A and C, while A and C are not clearly different. If you only care about comparing each treatment with a control, Dunnett's test is more powerful. If you planned a few specific contrasts in advance, test those directly.

Checking the assumptions

ANOVA assumes independent observations, roughly normal residuals and equal variances across groups. Check:

• Residuals vs fitted and a Q-Q plot, as in the diagnostics lesson.
• Group standard deviations: a ratio of largest to smallest SD below about 2 is usually fine, especially with equal group sizes.
• If variances clearly differ, use Welch's ANOVA (pingouin.welch_anova or statsmodels.stats.oneway.anova_oneway(..., use_var="unequal")).
• For badly skewed data with small groups, the rank-based Kruskal–Wallis test is an alternative, though it tests a slightly different hypothesis.

Balanced designs (equal n per group) make ANOVA much more robust to unequal variances, one more reason to plan the experiment rather than take what comes.

Common mistakes

• Running all pairwise t-tests without correction. With 5 groups there are 10 pairs, and the chance of at least one false positive at 5% each is about 40%.
• Concluding "no difference" from a non-significant F. It may mean the experiment was too small. Report confidence intervals for the differences.
• Ignoring the block you designed in. If the experiment was blocked, the model must include the block.
• Treating repeated measurements as replicates, which shrinks the error term and inflates F.

Quick recap

• ANOVA splits total variation into between-group and within-group parts.
• F = MS_between / MS_within; compare with F(k − 1, N − k).
• A significant F needs a follow-up (Tukey, Dunnett, planned contrasts).
• ANOVA is a linear model, so diagnose residuals and include design terms such as blocks.

Try it

In the Do tab you will compute the ANOVA table for the catalyst data from scratch with the standard library and confirm the F statistic of 21.5.