Point Estimation and Confidence Intervals

Maximum likelihood estimation

Maximum likelihood estimation (MLE) chooses the parameter value under which the observed sample is most probable. For independent observations x_1,\dots,x_n from a density f(x;\theta), the likelihood is L(\theta)=\prod_{i=1}^{n} f(x_i;\theta), and the MLE \hat\theta maximizes L(\theta) over the admissible parameter space. Since \ln is increasing, one usually maximizes \ell(\theta)=\ln L(\theta) and solves the likelihood equation \ell'(\theta)=0.

Two situations are frequently tested. First, when the parameter space is restricted, the stationary point may lie outside it. If \ell is monotone on the admissible set, the maximum occurs at an endpoint, and the MLE is the bound nearest to the unrestricted solution. Second, when the support of the density depends on the parameter, differentiation is not valid. One then examines L(\theta) directly. For a density on (0,\theta), the likelihood is zero unless \theta is at least the largest observation, and it decreases beyond that point, so the MLE is the largest observation.

Invariance: if \hat\theta is the MLE of \theta, then g(\hat\theta) is the MLE of g(\theta) for any function g, including quantiles and medians. A question about a population quantile therefore reduces to finding the MLE of the underlying parameter and substituting it.

When the density is zero below a location parameter \mu, the likelihood increases in \mu up to the smallest observation, so the MLE of \mu is the sample minimum, not the sample mean. When a variable is a monotone transform of an exponential variable, the likelihood can be written in the transformed variable, and the MLE of the mean is the average of the transformed observations.

For a density that changes form at a cut point, the log-likelihood splits into sums over the two regions. One still sets the derivative to zero and checks the second-order condition. For a Gamma sample with known shape \alpha and unknown rate \lambda, the MLE is \hat\lambda=\alpha n/\sum x_i. Because the reciprocal of a Gamma variable has a finite mean, the bias of this MLE can be computed exactly.

For a one-sided test about an exponential scale, the likelihood ratio is a monotone function of the sum of observations. This gives a uniformly most powerful test, and a chi-square pivot built from the sum yields an exact critical value. Two-parameter families such as the Gamma distribution are often estimated by the method of moments instead, which is treated later in this chapter. The sample minimum of a shifted distribution is both a natural point estimate and the statistic from which exact confidence limits are built.

Sufficiency and completeness

A statistic T=T(X_1,\dots,X_n) is sufficient for \theta when the conditional distribution of the sample given T does not depend on \theta. The factorization theorem gives the practical test: T is sufficient if and only if f(x;\theta)=g\bigl(T(x);\theta\bigr)\,h(x), where h does not involve \theta. Any factor that depends on the data but not on \theta can be absorbed into h.

Sufficient statistics are not unique. A statistic is minimal sufficient when the ratio f(x;\theta)/f(y;\theta) is free of \theta exactly when T(x)=T(y). For independent identically distributed samples, the order statistics are sufficient, and any statistic that is a one-to-one function of them is also minimal sufficient. A one-to-one transformation of a minimal sufficient statistic remains minimal sufficient.

In discrete families, sufficiency can be checked by asking whether the data determine the quantities that enter the likelihood. For Bernoulli samples, the probability mass depends on the data only through the number of successes. Any statistic that determines this number is sufficient, whereas a statistic that combines components and loses the count is not. For binomial variables with a common success probability, the total is sufficient, and conditional on the total the components do not involve p.

A statistic T is complete if E_\theta[k(T)]=0 for all \theta implies k(T)=0 almost surely. A complete sufficient statistic is minimal sufficient, and by the Lehmann-Scheffé theorem, an unbiased function of it is the unique UMVUE. Regular exponential families with an open natural parameter set are complete. For a Poisson sample, the total is complete and sufficient.

Curved exponential families are sufficient but not complete. When the mean and the standard deviation of a normal law are tied to the same parameter, as in N(\theta,\theta^2), a nontrivial function of the sufficient pair can have zero expectation for every \theta, so completeness fails.

A family has the monotone likelihood ratio (MLR) property in T when f(x;\theta_2)/f(x;\theta_1) is a monotone function of T for \theta_2>\theta_1. This property yields monotone critical regions and uniformly most powerful one-sided tests.

An ancillary statistic has a distribution that does not depend on \theta. For a uniform sample on a shifted interval, the range has such a distribution, so it carries no information about the location.

Unbiasedness is not preserved by nonlinear transformations. The sample mean and the sample variance of a Poisson sample are unbiased for \lambda. However, the square root of the sample variance is biased for \sqrt{\lambda}, because by Jensen's inequality E\sqrt{S^2}<\sqrt{E S^2} for a non-degenerate variable. Consistency is also a separate property from unbiasedness.

When an unbiased estimator must depend on the data only through a statistic such as X_{(2)}, unbiasedness becomes an integral equation in \theta. Differentiating with respect to \theta gives the estimator explicitly.

Confidence intervals for normal parameters

A confidence interval for a normal mean is built from a pivot with a known distribution. When \sigma is known, \sqrt{n}(\bar X-\mu)/\sigma is standard normal. When \sigma is unknown, \sqrt{n}(\bar X-\mu)/S follows a t distribution with n-1 degrees of freedom.

A 95% confidence interval is not unique. Any pair of quantiles that leaves total tail probability 0.05 gives a valid interval, and the symmetric interval is only the conventional choice. This holds whether \sigma is known or unknown.

For a normal mean with known variance, the two-sided test based on z=(\bar x-\mu_0)/(\sigma/\sqrt{n}) is the uniformly most powerful unbiased test. The two-sided p-value is 2\{1-\Phi(|z|)\}, so a statistic of magnitude 1.96 gives a p-value of about 0.05.

For a one-sided test of size \alpha with Type II error at most \beta, the sample size satisfies n \ge \left[(z_\alpha+z_\beta)\sigma/(\mu_1-\mu_0)\right]^2, where z_p is the upper p point of the standard normal. The two critical values are fixed by the two error levels, and n is the smallest integer meeting the inequality.

The power of a two-sided test at an alternative mean is the sum of the two tail probabilities of the sample mean beyond the critical limits, each standardized using the alternative mean. Adding both tails is essential, because the smaller tail is not negligible when the alternative is close to the null.

Cramér-Rao lower bound

For a regular family with per-observation Fisher information I(\theta)=E\left[\left(\partial \ln f/\partial\theta\right)^2\right], the Cramér-Rao lower bound for an unbiased estimator of \tau(\theta) based on n observations is \{\tau'(\theta)\}^2/(nI(\theta)). For \tau(\theta)=\theta, it reduces to 1/(nI(\theta)). The bound applies only to unbiased estimators; a biased estimator may have smaller variance and is not covered.

An unbiased estimator that attains the bound is called efficient. Attainment is possible only when the estimator is a linear function of the score, which is typical for exponential families.

For a normal mean with known variance \sigma^2, the Fisher information per observation is 1/\sigma^2, so the bound for unbiased estimators of the mean is \sigma^2/n, and the sample mean attains it.

For a Gamma distribution with known shape \alpha and scale \beta, the per-observation information about \beta is \alpha/\beta^2. For a function \tau(\beta), the bound is obtained by multiplying through by \tau'(\beta)^2.

When the mean is known, the variance estimator \frac{1}{n}\sum(X_i-\mu)^2 is unbiased for \sigma^2 and has variance 2\sigma^4/n, which equals the bound. This estimator is efficient even though the sample variance about the sample mean would have a different variance.

The bound also applies to unbiased estimators of a function of the parameter. If g(\theta) is the sum of expected values of independent normal observations, then the bound is \{g'(\theta)\}^2/(nI(\theta)), and an unbiased linear combination of the observations can attain it.

Method of moments estimation

The method of moments estimates parameters by equating population moments E(X^k) to sample moments \frac{1}{n}\sum X_i^k and solving the resulting equations. A single parameter needs one equation; k parameters need the first k moments.

If the population mean is a multiple c\theta, then equating c\theta with \bar X gives \hat\theta=\bar X/c. The factor is essential, because forgetting it gives an estimator that estimates c\theta instead of \theta.

For a density on (0,\theta) whose mean is \theta/3, the method of moments estimator is 3\bar X. Substituting the observed sample mean gives the estimate.

When the population moment is a function of the parameter through a power, one first computes the moment of the transformed variable and then solves the equation for the positive root. For an exponential variable with mean \lambda, the second moment is 2\lambda^2, so the estimator of \lambda is the positive square root of the sample average of the squared values divided by 2.