A/B test design — Data Analytics in Practice

What you'll learn

Size and plan an A/B test
Write an experiment plan with hypothesis, unit, primary and guardrail metrics and a decision rule
Explain how significance level, power and minimum detectable effect drive sample size
Convert a sample size into a test duration in full weeks

Plan an A/B test with a clear hypothesis, primary metric and sample size before it starts.

Why it matters

An A/B test (a randomised controlled experiment) shows a random half of users the current version (A, the control) and the other half a change (B, the treatment), then compares outcomes. Because assignment is random, the only systematic difference between the groups is the change itself, so a difference in outcomes can be attributed to it. That is the strongest evidence of cause an analyst usually gets.

Most failed experiments fail in the planning, not the analysis: no clear metric, too few users to detect anything, or a test stopped the moment it looked good. This lesson covers the plan.

The experiment plan

Write this down and get agreement before launch:

Hypothesis. "Showing delivery estimates at checkout will increase purchase conversion, because uncertainty about delivery causes drop-off."
Unit of randomisation. Usually the user (not the page view), so one person always sees the same version.
Primary metric. One metric that decides the test, defined precisely (Module 1): checkout conversion per user.
Guardrail metrics. Things that must not get worse: refund rate, page load time, support contacts.
Minimum detectable effect (MDE). The smallest change worth detecting, decided from the business case: "anything below +1 point is not worth the engineering cost."
Significance level and power. Usually α = 0.05 and power = 0.80.
Sample size and duration, calculated from the above.
Decision rule. "Ship if the primary metric improves significantly and no guardrail worsens significantly."

Significance, power and errors

A false positive (Type I error) is concluding the change works when it does not. Its probability is α, the significance level, commonly 5%.
A false negative (Type II error) is missing a real effect. Its probability is β. Power = 1 − β, the chance of detecting a real effect of size MDE. 80% is common.

Lower α or higher power both require more users. So does a smaller MDE, and dramatically: halving the MDE roughly quadruples the sample size, because n scales with 1 / MDE².

Worked example: sample size for a conversion rate

Baseline conversion is 10%. The team cares about an increase to 12% (MDE = 2 percentage points). With α = 0.05 (two-sided) and 80% power, the standard formula for two proportions is:

n per group = ( z_(1-α/2) · sqrt(2 p̄ (1 - p̄)) + z_(1-β) · sqrt(p1(1-p1) + p2(1-p2)) )² / (p2 - p1)²

with p1 = 0.10, p2 = 0.12, p̄ = 0.11, z_(0.975) ≈ 1.96 and z_(0.80) ≈ 0.84. That gives about 3,841 users per group, 7,682 in total.

If 500 eligible users visit checkout per day, the test needs about 16 days. Round up to 21 days (three full weeks), so every weekday appears equally often: behaviour on Mondays and Saturdays often differs.

Online calculators (Evan Miller's sample size calculator is widely used) and statsmodels.stats.power in Python give the same numbers. Commercial experimentation platforms do this for you, but you should still understand the inputs.

Practical design choices

Relative vs absolute MDE. "+2 points" (10% to 12%) and "+20% relative" are the same here. Say which you mean.
Low traffic? Test bolder changes (larger MDE), use a metric closer to the change (clicks on the delivery estimate rather than revenue), or accept a longer test.
Split. 50/50 is the most efficient; uneven splits need more total users.
Check eligibility. Only users who could see the change (those who reach checkout) should be in the test; including everyone dilutes the effect.
A/A tests (both groups get the same version) are a useful check that the setup and analysis produce no false differences.

Common mistakes

Choosing the MDE after seeing results.
Too many primary metrics; with ten metrics, one will likely be "significant" by chance.
Randomising by page view, so users flip between versions.
Running only on weekdays or during a promotion that will not repeat.

Quick recap

Write the plan first: hypothesis, unit, primary and guardrail metrics, MDE, α, power, sample size, decision rule.
Sample size grows with smaller MDE (1/MDE²), lower α and higher power.
Convert sample size to duration using eligible daily traffic and round to full weeks.

Try it

In the Do tab, implement the sample-size formula and turn it into a test duration.

Key terms (6)

A/B test: A randomised controlled experiment that shows a control version to some users and a changed version to others and compares outcomes.
Minimum detectable effect (MDE): The smallest change worth detecting, chosen from the business case before the test starts.
false positive: Concluding a change works when it does not (Type I error); its probability is the significance level alpha.
Power: 1 - beta: the chance of detecting a real effect of the MDE's size; 80% is common.
Guardrail metrics: Metrics that must not get worse during a test, such as refund rate or page load time.
A/A tests: Tests where both groups get the same version, used to check that the setup produces no false differences.