Priors and posteriors

Why this matters

The previous lesson updated a single yes/no hypothesis. Most real questions are about an unknown rate: a conversion rate, a defect rate, a cure rate, a claim frequency. Bayesian inference gives you a full probability distribution for that rate, so you can answer the questions decision-makers actually ask: "What is the chance version B is better than A?" or "How likely is the defect rate above 2%?" The beta-binomial model is the cleanest example, and it is used directly in A/B testing tools, reliability work and actuarial credibility theory.

The beta distribution

The beta distribution lives on the interval 0 to 1, which makes it a natural model for a rate p. It has two positive parameters, a and b:

mean = a / (a + b)
var  = a b / [ (a + b)² (a + b + 1) ]

A helpful reading is "a prior successes and b prior failures":

• Beta(1, 1): flat (uniform). Every rate from 0 to 1 is equally plausible.
• Beta(2, 8): centred near 0.2, fairly vague (like 10 observations' worth of information).
• Beta(20, 80): also centred at 0.2, much more confident (100 observations' worth).

The conjugate update

With a Beta(a, b) prior and s successes in n trials (binomial likelihood), the posterior is again a beta:

posterior = Beta(a + s, b + n - s)

That is the whole calculation: add successes to a and failures to b. A prior and likelihood that give a posterior of the same family are called conjugate.

The posterior mean is a weighted compromise between the prior mean and the observed rate:

posterior mean = (a + s) / (a + b + n)

With little data the prior dominates; with lots of data the observed rate dominates. Actuaries call this the credibility weighting.

Worked example: a sign-up rate

A new landing page converted 18 of 100 visitors.

Flat prior Beta(1, 1): posterior Beta(19, 83), mean 19 / 102 ≈ 0.186. A 95% credible interval (the middle 95% of the posterior) is about 0.117 to 0.267.

Informed prior from past pages, Beta(20, 80) (mean 0.20): posterior Beta(38, 162), mean 0.19, and a narrower interval because the prior adds information.

A credible interval has the reading people often (wrongly) give to confidence intervals: "given the model and prior, there is a 95% probability the rate lies in this range".

Summaries in code

import numpy as np
rng = np.random.default_rng(0)
draws = rng.beta(19, 83, size=100_000)
print(draws.mean())                         # about 0.186
print(np.percentile(draws, [2.5, 97.5]))    # about [0.117, 0.267]
print((draws > 0.25).mean())                # P(rate > 25%), about 0.05

With SciPy you can get exact values instead: scipy.stats.beta(19, 83).ppf([0.025, 0.975]).

Worked example: an A/B test

Version A converted 18 of 100 visitors; version B converted 30 of 100. With flat priors:

A: Beta(19, 83)        B: Beta(31, 71)

What is P(p_B > p_A)? There is no simple formula, but sampling makes it easy: draw from both posteriors and count how often B's draw is larger.

a = rng.beta(19, 83, size=100_000)
b = rng.beta(31, 71, size=100_000)
print((b > a).mean())          # about 0.976

"There is about a 98% probability B has the higher conversion rate" is a statement product managers can act on. You can also look at the distribution of b - a to see how much better B is likely to be, which matters more than whether it is better at all.

Choosing a prior

• Use a weakly informative prior (such as Beta(1, 1) or Beta(2, 2)) when you know little.
• Use an informed prior from historical data when you have it, and say where it came from.
• Run a sensitivity check: refit with a different reasonable prior. If the conclusion flips, the data is not yet strong enough to decide.

Common mistakes

• Mixing up a and b. a counts successes, b failures.
• Adding trials instead of failures to b. The update is b + (n - s).
• A prior that is too strong. Beta(2000, 8000) will barely move with 100 new visitors. Make sure the prior's "worth" in observations is honest.
• Reading P(B > A) = 0.98 as "B is 98% better". It is the probability B is better at all, not the size of the improvement.

Recap

• Beta prior + binomial data -> beta posterior: Beta(a + s, b + n - s).
• Posterior mean = (a + s) / (a + b + n), a compromise between prior and data.
• Credible intervals and probabilities like P(B > A) come straight from posterior draws.
• Check how sensitive conclusions are to the prior.

Try it

In the Do tab, write the beta-binomial update and estimate P(B > A) by sampling with NumPy.