Experiment pitfalls

Why it matters

An A/B test is only as trustworthy as the way it was run and read. Experimentation teams at large companies report that many apparent wins disappear on re-test, often because of a handful of well-known mistakes. Knowing them lets you challenge a result before it becomes a decision, including your own results.

Peeking (optional stopping)

The problem: checking the results every day and stopping as soon as p < 0.05. A p-value is designed to be checked once, at the planned sample size. If you look repeatedly, random fluctuations will cross the 0.05 line at some point far more often than 5% of the time. Simulations show that checking daily over a month can push the false positive rate well above 20%.

Example: on day 3 the treatment shows +8% with p = 0.04. The team ships. By the planned end date, the effect would have settled near zero.

Fixes:

• Fix the sample size in advance and analyse once at the end.
• If you must monitor, use a method designed for it: sequential testing (with adjusted thresholds such as alpha spending) or "always-valid" p-values, which many experimentation platforms offer.
• Monitoring guardrails for harm (crashes, revenue collapse) is fine; stopping early for harm is sensible. Stopping early for a "win" is the problem.

Multiple comparisons

The problem: test 20 metrics, or 20 user segments, at α = 0.05 and you expect about one false positive even if nothing changed. "It did not move conversion overall, but it worked for Android users in Canada on weekends" is usually noise.

Fixes: one pre-registered primary metric; treat segment results as hypotheses for a new test; apply a correction (Bonferroni divides α by the number of tests; Benjamini-Hochberg controls the false discovery rate) when you genuinely need many comparisons.

Novelty and primacy effects

Novelty effect: users click a new feature because it is new, and the lift fades after a week or two. Primacy (change aversion) effect: users initially resist a redesign and perform worse, then adapt.

Fixes: run for at least one to two full weeks; plot the daily treatment effect over time and check whether it is shrinking or growing; compare new users (who have no old habits) with returning users.

Sample ratio mismatch

Covered in the previous lesson: if the split is not what was configured, something is broken, often in a way that biases results. Always check it.

Interference between groups

Randomisation assumes one user's experience does not affect another's. That breaks in:

• Marketplaces: a discount for treatment buyers can use up limited supply that control buyers would have bought.
• Social features: a treatment user's shared post reaches control users.

Fixes: randomise by cluster (city, region, time window) instead of by user, and accept the need for more data.

Other common traps

• Underpowered tests. With too few users, only huge effects reach significance, so the "wins" you do see tend to overstate the true effect (sometimes called the winner's curse).
• Changing the test mid-flight. Altering the treatment, traffic split or metric during the test mixes different experiments together.
• Seasonality and special events. A test during a holiday sale may not generalise to normal weeks.
• Simpson's paradox. If traffic mix differs over time (for example, ramping treatment from 10% to 50% during a week when weekend users behave differently), pooled results can point the opposite way from every day's result. Keep allocation constant, or analyse within periods.
• Twyman's law: "any figure that looks interesting or different is usually wrong." A +40% lift on a button colour should trigger a bug hunt before a celebration.

A pre-analysis checklist

1. Was the plan (metric, MDE, sample size, decision rule) written before launch?
2. Did the test run to its planned size without early stopping?
3. Does the sample ratio match the configured split?
4. Is the effect stable over time, or decaying?
5. Was there one primary metric, with segments treated as exploratory?
6. Did anything else change during the test (releases, campaigns, outages)?
7. Is the effect plausible in size?

Quick recap

• Do not stop a test the moment it looks significant; use the planned size or a sequential method.
• One primary metric; segment "wins" are new hypotheses.
• Watch for novelty and change-aversion by plotting the effect over time.
• Check SRM and interference, and be suspicious of surprisingly large effects.