Priors and posteriors

Why this matters

The previous lesson updated a single yes/no hypothesis. Most real questions are about an unknown rate: a conversion rate, a defect rate, a cure rate, a claim frequency. Bayesian inference gives you a full probability distribution for that rate, so you can answer the questions decision-makers actually ask: "What is the chance version B is better than A?" or "How likely is the defect rate above 2%?" The beta-binomial model is the cleanest example, and it is used directly in A/B testing tools, reliability work and actuarial credibilityThe actuarial name for weighting the prior mean against the observed rate according to how much data there is. theory.

The beta distribution

The beta distributionA distribution on 0 to 1 with parameters a and b, read as a prior successes and b prior failures; mean a/(a + b). lives on the interval 0 to 1, which makes it a natural model for a rate p. It has two positive parameters, a and b:

mean = a / (a + b)
var  = a b / [ (a + b)² (a + b + 1) ]

A helpful reading is "a prior successes and b prior failures":

• Beta(1, 1): flat (uniform). Every rate from 0 to 1 is equally plausible.
• Beta(2, 8): centred near 0.2, fairly vague (like 10 observations' worth of information).
• Beta(20, 80): also centred at 0.2, much more confident (100 observations' worth).

The conjugate update

With a Beta(a, b) prior and s successes in n trials (binomial likelihood), the posterior is again a beta:

posterior = Beta(a + s, b + n - s)

That is the whole calculation: add successes to a and failures to b. A prior and likelihood that give a posterior of the same family are called conjugateA prior and likelihood that give a posterior in the same family, like a beta prior with binomial data..

The posterior mean is a weighted compromise between the prior mean and the observed rate:

posterior mean = (a + s) / (a + b + n)

With little data the prior dominates; with lots of data the observed rate dominates. Actuaries call this the credibility weighting.

Worked example: a sign-up rate

A new landing page converted 18 of 100 visitors.

Flat prior Beta(1, 1): posterior Beta(19, 83), mean 19 / 102 ≈ 0.186. A 95% credible intervalA range containing a stated share (such as 95%) of the posterior probability for a parameter. (the middle 95% of the posterior) is about 0.117 to 0.267.

Informed prior from past pages, Beta(20, 80) (mean 0.20): posterior Beta(38, 162), mean 0.19, and a narrower interval because the prior adds information.

A credible interval has the reading people often (wrongly) give to confidence intervals: "given the model and prior, there is a 95% probability the rate lies in this range".

Summaries in code

import numpy as np
rng = np.random.default_rng(0)
draws = rng.beta(19, 83, size=100_000)
print(draws.mean())                         # about 0.186
print(np.percentile(draws, [2.5, 97.5]))    # about [0.117, 0.267]
print((draws > 0.25).mean())                # P(rate > 25%), about 0.05

With SciPy you can get exact values instead: scipy.stats.beta(19, 83).ppf([0.025, 0.975]).

Worked example: an A/B test

Version A converted 18 of 100 visitors; version B converted 30 of 100. With flat priors:

A: Beta(19, 83)        B: Beta(31, 71)

What is P(p_B > p_A)? There is no simple formula, but sampling makes it easy: draw from both posteriors and count how often B's draw is larger.

a = rng.beta(19, 83, size=100_000)
b = rng.beta(31, 71, size=100_000)
print((b > a).mean())          # about 0.976

"There is about a 98% probability B has the higher conversion rate" is a statement product managers can act on. You can also look at the distribution of b - a to see how much better B is likely to be, which matters more than whether it is better at all.

Choosing a prior

• Use a weakly informativeA prior such as Beta(1, 1) or Beta(2, 2) that carries little information, used when you know little. prior (such as Beta(1, 1) or Beta(2, 2)) when you know little.
• Use an informed prior from historical data when you have it, and say where it came from.
• Run a sensitivity check: refit with a different reasonable prior. If the conclusion flips, the data is not yet strong enough to decide.

Common mistakes

• Mixing up a and b. a counts successes, b failures.
• Adding trials instead of failures to b. The update is b + (n - s).
• A prior that is too strong. Beta(2000, 8000) will barely move with 100 new visitors. Make sure the prior's "worth" in observations is honest.
• Reading P(B > A) = 0.98 as "B is 98% better". It is the probability B is better at all, not the size of the improvement.

Recap

• Beta prior + binomial data -> beta posterior: Beta(a + s, b + n - s).
• Posterior mean = (a + s) / (a + b + n), a compromise between prior and data.
• Credible intervals and probabilities like P(B > A) come straight from posterior draws.
• Check how sensitive conclusions are to the prior.

Try it

In the Do tab, write the beta-binomial update and estimate P(B > A) by sampling with NumPy.