Apply the three principles of experimental design and lay out a randomised complete block experiment.
Why this matters
Module 1 analysed data that already existed. This module is about collecting data so that the analysis is easy and the conclusion is trustworthy. A well-designed experiment of 16 runs often tells you more than 1,000 rows of historical process data, because you chose the settings and controlled how they were assigned. R. A. Fisher put it bluntly in 1938: calling in a statistician after the experiment is done is often asking for a post-mortem. The three principles below, randomisation, replication and blocking, are what make an experiment defensible to a reviewer, a regulator or a sceptical plant manager.
Vocabulary
- Experimental unit: the thing a treatment is applied to independently (a batch, a plot, a mouse, a wafer, a customer).
- Factor: an input you deliberately set (temperature, supplier, dose). Its settings are levels.
- Treatment: one combination of factor levels.
- Response: what you measure.
- Nuisance factor: something that affects the response but is not of interest (day, operator, raw-material lot, instrument).
Randomisation
Assign treatments to units, and order the runs, at random. This protects you against things you did not think of. If you run all the high-temperature batches in the morning and the low ones in the afternoon, any morning-versus-afternoon difference (ambient humidity, a tired operator, a warming instrument) is mixed up with temperature, and no analysis can separate them.
Randomisation does not make groups identical; it makes the leftover differences chance differences, which is exactly what the error term and the p-value account for. In practice:
import random
random.seed(2026) # record the seed in your lab notebook
runs = ["low"] * 4 + ["high"] * 4
random.shuffle(runs)
print(runs) # e.g. ['high', 'low', 'low', 'high', ...]
Keep the seed and the resulting run sheet; it is part of the method.
Replication
Replication means applying the same treatment to more than one independent experimental unit. It gives you an estimate of pure error, without which you cannot tell a real effect from noise, and more replicates give more precision (the standard error of a mean falls with 1/sqrt(n)).
The trap is pseudo-replication: measuring the same unit several times and treating the measurements as independent. Three readings from one batch tell you about measurement error, not batch-to-batch variation. If treatments are applied to batches, the batch is the unit, and n is the number of batches.
Blocking
A block is a group of units that are more alike than the units overall: runs on the same day, specimens cut from the same sheet, patients at the same site. Blocking means running every treatment within each block and comparing treatments inside blocks. Block-to-block differences then cancel out of the treatment comparisons and are removed from the error term, which makes the experiment more sensitive.
The rule of thumb, often attributed to George Box: "Block what you can, randomise what you cannot."
Worked example: a randomised complete block design
A lab wants to compare four buffer recipes (A, B, C, D) on enzyme activity. Only four assays fit on one plate reader run per day, and day-to-day drift in the reader is known to be large. Design:
- Blocks: 4 days. Treatments: 4 buffers. Each buffer appears once per day.
- Randomisation: within each day, the order (and plate position) of the four buffers is shuffled separately.
run 1 run 2 run 3 run 4
Monday C A D B
Tuesday B D A C
Wednesday A C B D
Thursday D B C A
The analysis model is activity ~ C(buffer) + C(day). The day term soaks up the drift, and the buffer comparison has (4 − 1) × (4 − 1) = 9 error degrees of freedom. If you had ignored the days and randomised all 16 runs freely, the day-to-day variation would sit inside the error term and could hide a real difference between buffers.
Blocks are usually treated as fixed effects in a small design like this, or as random effects when there are many of them (module 3).
Choosing the number of runs
Size the experiment before you run it. You need three things: the smallest difference worth detecting (from the scientist, not the statistician), an estimate of the noise standard deviation (from past data or a pilot), and the desired power (usually 80–90%). The power ideas from Probability & Statistical Inference apply directly; statsmodels.stats.power has calculators for t-tests and ANOVA. If the answer is "more runs than we can afford", it is better to know before spending the budget.
Other designs you will meet
- Completely randomised design (CRD): no blocks; fine when units are homogeneous.
- Latin square: blocks in two directions at once (day and operator), each treatment once per row and column.
- Split-plot: some factors are hard to change (oven temperature), so they are randomised to large "whole plots" and easy factors to "subplots" inside them. Split-plots need a mixed-model analysis; analysing them as if fully randomised overstates precision for the hard-to-change factor.
- Crossover: each subject gets several treatments in random order, with washout periods.
Common mistakes
- Running treatments in a convenient order (all of A, then all of B). It confounds treatment with time.
- Calling repeated measurements replicates.
- Blocking but analysing as if unblocked, throwing away the precision you paid for.
- Changing the plan mid-experiment without recording it. Deviations are normal; undocumented ones are fatal.
Quick recap
- Randomise to protect against unknown nuisance variables.
- Replicate independent units to estimate error; beware pseudo-replication.
- Block known nuisance variables and compare treatments within blocks.
- Size the experiment with power before you start.
Try it
The Do tab walks you through designing a small blocked experiment on paper, from choosing the unit and blocks to producing a randomised run sheet.