Estimation and MLE

Why this matters

Module 1 started from known parameters ("defects occur at 2%", "calls arrive at 3 per hour") and computed probabilities. Real work runs the other way: you have data and need the parameter. What defect rate best explains today's inspections? What failure rate fits these component lifetimes? Maximum likelihood estimation (MLE) is the standard answer. It is how logistic regression, most machine-learning loss functions, survival models and actuarial frequency models are fitted.

Estimators and their qualities

An estimator is a rule that turns data into a guess for a parameter, such as the sample mean x̄ for the population mean μ. Two properties matter most:

• Bias: on average over many samples, does the estimator hit the true value? The sample variance with n - 1 is unbiased; with n it is slightly biased low.
• Variance: how much does it bounce around from sample to sample? Its square root is the standard error.

A good estimator has both small. Their combined effect is the mean squared error: MSE = bias² + variance.

Likelihood

The likelihood L(θ) is the probability (or density) of the data you actually observed, viewed as a function of the parameter θ. For independent observations it is a product:

L(θ) = P(x1 | θ) × P(x2 | θ) × ... × P(xn | θ)

The maximum-likelihood estimate θ̂ is the value of θ that makes the observed data most probable.

Products of many small numbers underflow on a computer, so in practice we maximise the log-likelihood, a sum:

ℓ(θ) = log L(θ) = sum of log P(xi | θ)

The log is increasing, so the same θ maximises both.

Worked example: a coin or a pass rate

You observe 7 successes in 10 trials. Under a binomial model,

ℓ(p) = 7 log p + 3 log(1 - p) + constant

Try a few values: ℓ(0.5) ≈ -6.93, ℓ(0.7) ≈ -6.11, ℓ(0.9) ≈ -7.64 (dropping the constant). The peak is at p̂ = 0.7. Setting the derivative to zero confirms it: 7/p - 3/(1-p) = 0 gives p = 7/10. The MLE is the sample proportion, which is reassuring.

Worked example: Poisson defects per day

Daily defect counts over 8 days: 2, 3, 1, 0, 4, 2, 3, 1. For a Poisson model with rate λ,

log P(k | λ) = k log λ - λ - log(k!)
ℓ(λ) = (sum of k) log λ - n λ - sum of log(k!)

Setting the derivative to zero gives λ̂ = (sum of k) / n = x̄ = 16 / 8 = 2.0. Again the MLE is the sample mean.

You do not always get a closed form, so it is worth knowing how to find the peak numerically. A grid search is the simplest method:

import math

data = [2, 3, 1, 0, 4, 2, 3, 1]

def loglik(lam):
    return sum(k * math.log(lam) - lam - math.lgamma(k + 1) for k in data)

grid = [i / 100 for i in range(1, 801)]     # 0.01 to 8.00
best = max(grid, key=loglik)
print(best)                                  # 2.0

math.lgamma(k + 1) is log(k!) computed without overflow.

Common closed-form MLEs

The last row is a useful reminder that MLEs are not always unbiased. They are, however, consistent (they converge to the truth as n grows) and, for large samples, approximately normal with the smallest possible variance. That is why they are the default.

In libraries

Real models have several parameters, and you maximise with an optimiser instead of a grid. A typical pattern is to minimise the negative log-likelihood:

from scipy.optimize import minimize_scalar
res = minimize_scalar(lambda lam: -loglik(lam), bounds=(0.01, 10), method="bounded")
print(res.x)        # about 2.0

scipy.stats distributions also have a .fit(data) method that returns MLEs directly, and machine-learning "log loss" or "cross-entropy" is exactly a negative log-likelihood.

Common mistakes

• Multiplying probabilities instead of adding logs, which underflows to 0 for large data.
• Searching outside the valid range. log(0) fails; keep p in (0, 1) and λ > 0.
• A grid that is too coarse or too narrow. If the best value sits at the edge of the grid, widen it.
• Treating the MLE as the truth. It is an estimate with a standard error; report one.

Recap

• Likelihood: probability of the observed data as a function of the parameter.
• Maximise the log-likelihood (a sum) instead of the likelihood (a product).
• Many MLEs are familiar: the sample proportion, the sample mean, 1 / x̄.
• When no formula exists, search numerically (grid search, then an optimiser).

Try it

In the Do tab, write a Poisson log-likelihood and find its maximum by grid search.