Analysing a test

Why it matters

When an experiment finishes, stakeholders ask one question: "Did it work?" Answering well means more than reading a p-value off a tool. You need to check the test ran correctly, estimate how big the effect is with its uncertainty, decide using the rule agreed in the plan, and explain the result so a non-statistician understands what is known and what is not.

Step 1: Check the test before the result

• Sample ratio mismatch (SRM). With a 50/50 split, group sizes should be close to equal. 5,000 vs 5,040 is normal noise; 5,000 vs 5,400 is extremely unlikely by chance and points to a bug (for example, the new version crashes for some users before they are logged). A chi-square or z-test on the counts with a strict threshold (p < 0.001) is the usual check. If SRM fails, do not trust the results; find the cause.
• Duration and sample size match the plan.
• Guardrail and data checks: no tracking outages, no overlapping experiments that touch the same page.

Step 2: Estimate the effect

Suppose control converted 400 of 4,000 users (10.0%) and treatment 460 of 4,000 (11.5%).

• Absolute lift = 11.5% − 10.0% = +1.5 percentage points.
• Relative lift = 1.5 / 10.0 = +15%.

Always say which one you mean. "+15%" and "+1.5 points" describe the same result and are easy to confuse.

Step 3: Quantify uncertainty with a confidence interval

The standard error of the difference between two proportions (unpooled) is:

SE = sqrt( ra(1 - ra)/na + rb(1 - rb)/nb )
   = sqrt( 0.10·0.90/4000 + 0.115·0.885/4000 ) ≈ 0.00694

A 95% confidence interval is difference ± 1.96 × SE:

0.015 ± 1.96 × 0.00694  →  +0.14 to +2.86 percentage points

Interpretation: the data are consistent with a true lift anywhere from almost nothing to nearly 3 points. The interval excludes zero, so the result is significant at the 5% level. The width tells stakeholders how precise the estimate is, which a p-value alone does not.

Step 4: The p-value

The p-value answers: "If the change had no effect, how often would we see a difference at least this large by chance?" For proportions, a two-sample z-test uses the pooled rate (because the null hypothesis says both groups share one rate):

p = (400 + 460) / 8000 = 0.1075
SE_pooled = sqrt( p(1 - p)(1/4000 + 1/4000) ) ≈ 0.00693
z = 0.015 / 0.00693 ≈ 2.17   →   two-sided p ≈ 0.030

Since 0.030 < 0.05, we reject the null hypothesis. In Python on your own machine, statsmodels.stats.proportion.proportions_ztest gives the same z and p.

What the p-value is not: it is not the probability that the change works, and it says nothing about the size of the effect. A tiny, useless effect can be highly significant with millions of users.

Step 5: Decide and report

Apply the decision rule from the plan. A clear write-up:

Delivery estimates at checkout: ship. Conversion rose from 10.0% to 11.5% (+1.5 points, +15% relative; 95% CI +0.1 to +2.9 points; p = 0.03), over 21 days and 8,000 users. Guardrails (refunds, load time) showed no significant change. The lower end of the interval is small, so we will keep monitoring conversion for four weeks after launch.

Add the business translation: "+1.5 points at current traffic is about 230 extra orders a month."

When results are not significant

"Not significant" does not mean "no effect". It means the data cannot distinguish the effect from zero. Report the interval: "between −0.8 and +2.1 points" tells the team a large positive effect is still possible and a large negative one is unlikely. Options: accept the change is at best small, or run a larger test if a small effect would still be valuable.

Common mistakes

• Skipping the SRM check.
• Reporting only "significant / not significant" without the effect size and interval.
• Mixing up relative and absolute lift.
• Analysing many segments after the fact and reporting only the one that "worked" (see the next lesson).
• Treating p = 0.051 and p = 0.049 as completely different worlds.

Quick recap

• Check SRM, duration and data quality first.
• Report absolute and relative lift with a confidence interval.
• The p-value measures surprise under "no effect", not the size or probability of an effect.
• Decide by the pre-agreed rule and translate the result into business terms.

Try it

In the Do tab, write the analysis function: lifts, confidence interval, p-value and an SRM check.