Work like a mid-level data analyst: define KPIs, forecast, run and analyse A/B tests, model analytics tables and turn stakeholder requests into recommendations.

Data Analytics in Practice

## Why it matters

An **A/B test** (a randomised controlled experiment) shows a random half of users the current version (A, the control) and the other half a change (B, the treatment), then compares outcomes. Because assignment is random, the only systematic difference between the groups is the change itself, so a difference in outcomes can be attributed to it. That is the strongest evidence of cause an analyst usually gets.

Most failed experiments fail in the planning, not the analysis: no clear metric, too few users to detect anything, or a test stopped the moment it looked good. This lesson covers the plan.

## The experiment plan

Write this down and get agreement **before** launch:

1. **Hypothesis.** "Showing delivery estimates at checkout will increase purchase conversion, because uncertainty about delivery causes drop-off."
2. **Unit of randomisation.** Usually the user (not the page view), so one person always sees the same version.
3. **Primary metric.** One metric that decides the test, defined precisely (Module 1): checkout conversion per user.
4. **Guardrail metrics.** Things that must not get worse: refund rate, page load time, support contacts.
5. **Minimum detectable effect (MDE).** The smallest change worth detecting, decided from the business case: "anything below +1 point is not worth the engineering cost."
6. **Significance level and power.** Usually α = 0.05 and power = 0.80.
7. **Sample size and duration**, calculated from the above.
8. **Decision rule.** "Ship if the primary metric improves significantly and no guardrail worsens significantly."

## Significance, power and errors

- A **false positive** (Type I error) is concluding the change works when it does not. Its probability is **α**, the significance level, commonly 5%.
- A **false negative** (Type II error) is missing a real effect. Its probability is **β**. **Power** = 1 − β, the chance of detecting a real effect of size MDE. 80% is common.

Lower α or higher power both require more users. So does a smaller MDE, and dramatically: halving the MDE roughly **quadruples** the sample size, because n scales with 1 / MDE².

## Worked example: sample size for a conversion rate

Baseline conversion is 10%. The team cares about an increase to 12% (MDE = 2 percentage points). With α = 0.05 (two-sided) and 80% power, the standard formula for two proportions is:

```text
n per group = ( z_(1-α/2) · sqrt(2 p̄ (1 - p̄)) + z_(1-β) · sqrt(p1(1-p1) + p2(1-p2)) )² / (p2 - p1)²
```

with p1 = 0.10, p2 = 0.12, p̄ = 0.11, z_(0.975) ≈ 1.96 and z_(0.80) ≈ 0.84. That gives about **3,841 users per group**, 7,682 in total.

If 500 eligible users visit checkout per day, the test needs about 16 days. Round up to **21 days** (three full weeks), so every weekday appears equally often: behaviour on Mondays and Saturdays often differs.

Online calculators (Evan Miller's sample size calculator is widely used) and `statsmodels.stats.power` in Python give the same numbers. Commercial experimentation platforms do this for you, but you should still understand the inputs.

## Practical design choices

- **Relative vs absolute MDE.** "+2 points" (10% to 12%) and "+20% relative" are the same here. Say which you mean.
- **Low traffic?** Test bolder changes (larger MDE), use a metric closer to the change (clicks on the delivery estimate rather than revenue), or accept a longer test.
- **Split.** 50/50 is the most efficient; uneven splits need more total users.
- **Check eligibility.** Only users who could see the change (those who reach checkout) should be in the test; including everyone dilutes the effect.
- **A/A tests** (both groups get the same version) are a useful check that the setup and analysis produce no false differences.

## Common mistakes

- Choosing the MDE after seeing results.
- Too many primary metrics; with ten metrics, one will likely be "significant" by chance.
- Randomising by page view, so users flip between versions.
- Running only on weekdays or during a promotion that will not repeat.

## Quick recap

- Write the plan first: hypothesis, unit, primary and guardrail metrics, MDE, α, power, sample size, decision rule.
- Sample size grows with smaller MDE (1/MDE²), lower α and higher power.
- Convert sample size to duration using eligible daily traffic and round to full weeks.

## Try it

In the **Do** tab, implement the sample-size formula and turn it into a test duration.


Plan an A/B test with a clear hypothesis, primary metric and sample size before it starts.

A/B test design

AB Testing 101 | Fmr. Google Data Scientist Explains How to Calculate the Sample Size

Sample Size Estimation in A/B Tests Explained!

## Why it matters

When an experiment finishes, stakeholders ask one question: "Did it work?" Answering well means more than reading a p-value off a tool. You need to check the test ran correctly, estimate **how big** the effect is with its uncertainty, decide using the rule agreed in the plan, and explain the result so a non-statistician understands what is known and what is not.

## Step 1: Check the test before the result

- **Sample ratio mismatch (SRM).** With a 50/50 split, group sizes should be close to equal. 5,000 vs 5,040 is normal noise; 5,000 vs 5,400 is extremely unlikely by chance and points to a bug (for example, the new version crashes for some users before they are logged). A chi-square or z-test on the counts with a strict threshold (p < 0.001) is the usual check. If SRM fails, do not trust the results; find the cause.
- **Duration and sample size** match the plan.
- **Guardrail and data checks:** no tracking outages, no overlapping experiments that touch the same page.

## Step 2: Estimate the effect

Suppose control converted 400 of 4,000 users (10.0%) and treatment 460 of 4,000 (11.5%).

- **Absolute lift** = 11.5% − 10.0% = **+1.5 percentage points**.
- **Relative lift** = 1.5 / 10.0 = **+15%**.

Always say which one you mean. "+15%" and "+1.5 points" describe the same result and are easy to confuse.

## Step 3: Quantify uncertainty with a confidence interval

The standard error of the difference between two proportions (unpooled) is:

```text
SE = sqrt( ra(1 - ra)/na + rb(1 - rb)/nb )
   = sqrt( 0.10·0.90/4000 + 0.115·0.885/4000 ) ≈ 0.00694
```

A 95% confidence interval is difference ± 1.96 × SE:

```text
0.015 ± 1.96 × 0.00694  →  +0.14 to +2.86 percentage points
```

Interpretation: the data are consistent with a true lift anywhere from almost nothing to nearly 3 points. The interval excludes zero, so the result is significant at the 5% level. The width tells stakeholders how precise the estimate is, which a p-value alone does not.

## Step 4: The p-value

The p-value answers: "If the change had no effect, how often would we see a difference at least this large by chance?" For proportions, a two-sample z-test uses the **pooled** rate (because the null hypothesis says both groups share one rate):

```text
p = (400 + 460) / 8000 = 0.1075
SE_pooled = sqrt( p(1 - p)(1/4000 + 1/4000) ) ≈ 0.00693
z = 0.015 / 0.00693 ≈ 2.17   →   two-sided p ≈ 0.030
```

Since 0.030 < 0.05, we reject the null hypothesis. In Python on your own machine, `statsmodels.stats.proportion.proportions_ztest` gives the same z and p.

What the p-value is **not**: it is not the probability that the change works, and it says nothing about the size of the effect. A tiny, useless effect can be highly significant with millions of users.

## Step 5: Decide and report

Apply the decision rule from the plan. A clear write-up:

> **Delivery estimates at checkout: ship.** Conversion rose from 10.0% to 11.5% (+1.5 points, +15% relative; 95% CI +0.1 to +2.9 points; p = 0.03), over 21 days and 8,000 users. Guardrails (refunds, load time) showed no significant change. The lower end of the interval is small, so we will keep monitoring conversion for four weeks after launch.

Add the business translation: "+1.5 points at current traffic is about 230 extra orders a month."

## When results are not significant

"Not significant" does not mean "no effect". It means the data cannot distinguish the effect from zero. Report the interval: "between −0.8 and +2.1 points" tells the team a large positive effect is still possible and a large negative one is unlikely. Options: accept the change is at best small, or run a larger test if a small effect would still be valuable.

## Common mistakes

- Skipping the SRM check.
- Reporting only "significant / not significant" without the effect size and interval.
- Mixing up relative and absolute lift.
- Analysing many segments after the fact and reporting only the one that "worked" (see the next lesson).
- Treating p = 0.051 and p = 0.049 as completely different worlds.

## Quick recap

- Check SRM, duration and data quality first.
- Report absolute and relative lift with a confidence interval.
- The p-value measures surprise under "no effect", not the size or probability of an effect.
- Decide by the pre-agreed rule and translate the result into business terms.

## Try it

In the **Do** tab, write the analysis function: lifts, confidence interval, p-value and an SRM check.


Compute lift, a confidence interval and a p-value for an A/B test, and report the result honestly.

Analysing a test

A/B Testing Course 014: Statistical significance, confidence level and the certainty of your results

Confidence Interval [Simply explained]

## Why it matters

An A/B test is only as trustworthy as the way it was run and read. Experimentation teams at large companies report that many apparent wins disappear on re-test, often because of a handful of well-known mistakes. Knowing them lets you challenge a result before it becomes a decision, including your own results.

## Peeking (optional stopping)

**The problem:** checking the results every day and stopping as soon as p < 0.05. A p-value is designed to be checked **once**, at the planned sample size. If you look repeatedly, random fluctuations will cross the 0.05 line at some point far more often than 5% of the time. Simulations show that checking daily over a month can push the false positive rate well above 20%.

**Example:** on day 3 the treatment shows +8% with p = 0.04. The team ships. By the planned end date, the effect would have settled near zero.

**Fixes:**

- Fix the sample size in advance and analyse once at the end.
- If you must monitor, use a method designed for it: **sequential testing** (with adjusted thresholds such as alpha spending) or "always-valid" p-values, which many experimentation platforms offer.
- Monitoring guardrails for harm (crashes, revenue collapse) is fine; stopping early for harm is sensible. Stopping early for a "win" is the problem.

## Multiple comparisons

**The problem:** test 20 metrics, or 20 user segments, at α = 0.05 and you expect about one false positive even if nothing changed. "It did not move conversion overall, but it worked for Android users in Canada on weekends" is usually noise.

**Fixes:** one pre-registered primary metric; treat segment results as hypotheses for a new test; apply a correction (Bonferroni divides α by the number of tests; Benjamini-Hochberg controls the false discovery rate) when you genuinely need many comparisons.

## Novelty and primacy effects

**Novelty effect:** users click a new feature because it is new, and the lift fades after a week or two. **Primacy (change aversion) effect:** users initially resist a redesign and perform worse, then adapt.

**Fixes:** run for at least one to two full weeks; plot the daily treatment effect over time and check whether it is shrinking or growing; compare new users (who have no old habits) with returning users.

## Sample ratio mismatch

Covered in the previous lesson: if the split is not what was configured, something is broken, often in a way that biases results. Always check it.

## Interference between groups

Randomisation assumes one user's experience does not affect another's. That breaks in:

- **Marketplaces:** a discount for treatment buyers can use up limited supply that control buyers would have bought.
- **Social features:** a treatment user's shared post reaches control users.

**Fixes:** randomise by cluster (city, region, time window) instead of by user, and accept the need for more data.

## Other common traps

- **Underpowered tests.** With too few users, only huge effects reach significance, so the "wins" you do see tend to overstate the true effect (sometimes called the winner's curse).
- **Changing the test mid-flight.** Altering the treatment, traffic split or metric during the test mixes different experiments together.
- **Seasonality and special events.** A test during a holiday sale may not generalise to normal weeks.
- **Simpson's paradox.** If traffic mix differs over time (for example, ramping treatment from 10% to 50% during a week when weekend users behave differently), pooled results can point the opposite way from every day's result. Keep allocation constant, or analyse within periods.
- **Twyman's law:** "any figure that looks interesting or different is usually wrong." A +40% lift on a button colour should trigger a bug hunt before a celebration.

## A pre-analysis checklist

1. Was the plan (metric, MDE, sample size, decision rule) written before launch?
2. Did the test run to its planned size without early stopping?
3. Does the sample ratio match the configured split?
4. Is the effect stable over time, or decaying?
5. Was there one primary metric, with segments treated as exploratory?
6. Did anything else change during the test (releases, campaigns, outages)?
7. Is the effect plausible in size?

## Quick recap

- Do not stop a test the moment it looks significant; use the planned size or a sequential method.
- One primary metric; segment "wins" are new hypotheses.
- Watch for novelty and change-aversion by plotting the effect over time.
- Check SRM and interference, and be suspicious of surprisingly large effects.


Recognise peeking, multiple comparisons, novelty effects and other traps that make A/B results untrustworthy.

Experiment pitfalls

8 A/B testing mistakes every engineer should know

Peeking at A/B Tests: Why it matters, and what to do about it

Plan, size and analyse A/B tests, and avoid the mistakes that make experiment results untrustworthy.