Convergence and Limit Theorems

Almost sure convergence

Almost sure convergence means that X_n(\omega)\to X(\omega) for every sample point \omega outside a set of probability zero. An equivalent test is P(|X_n-X|>\varepsilon \text{ i.o.})=0 for every \varepsilon>0, where i.o. means infinitely often. The hierarchy to memorise is that almost sure convergence implies convergence in probability, which implies convergence in distribution. Convergence in r-th mean also implies convergence in probability. None of the converse implications holds in general.

The first and second Borel–Cantelli lemmas decide almost sure statements. If \sum_n P(A_n)<\infty, then P(A_n \text{ i.o.})=0. If the A_n are independent and \sum_n P(A_n)=\infty, then P(A_n \text{ i.o.})=1. For independent X_n that equal 1 with probability p_n and 0 otherwise, X_n\to0 almost surely exactly when \sum_n p_n<\infty. With p_n=n^{-2\alpha} this series converges precisely when \alpha>1/2. The admissible set of \alpha is therefore an open half-line whose infimum is 1/2.

The strong law of large numbers states that for i.i.d. X_i with finite mean \mu, \bar X_n\to\mu almost surely. The weak law gives only convergence in probability. Functions of such averages inherit the limit through the continuous mapping theorem. For example, if X is uniform on (0,2), write 2-X=2U with U uniform on (0,1). Since -\ln U is exponential with mean 1, the variable -\ln(2-X)=-\ln 2-\ln U has mean 1-\ln 2. The sample mean of these negated logarithms therefore converges almost surely to 1-\ln 2\approx 0.31.

For convergence in probability, take the shifted exponential with density e^{-(x-\theta)} for x\ge\theta. Its mean is \theta+1, so the sample mean converges in probability to \theta+1, not to half of it. For the minimum, P(\min>t)=e^{-n(t-\theta)} for t\ge\theta. Hence \min-\theta is exponential with mean 1/n, and E(\min)=\theta+1/n\to\theta. Limits of such statistics should be computed from the survival function, not guessed from a rule of thumb.

A distribution without a finite mean defeats the weak law. The standard Cauchy sample mean is again standard Cauchy for every n, so it does not converge. For any continuous distribution function F, however, F(X) is uniform on (0,1) by the probability integral transform. Its mean is 1/2, so the average of F(X_i) converges in probability to 1/2.

Almost sure limits pass through continuous functions and interact with independence, but they say nothing about moments. If X_n\to0 almost surely and the X_n are independent, then \sum_n P(|X_n|\ge\varepsilon)<\infty for each \varepsilon>0. Otherwise the second lemma would make these events occur infinitely often. A nondegenerate i.i.d. sequence cannot converge almost surely to a constant, because for some \varepsilon the independent events \{|X_n-c|>\varepsilon\} have a fixed positive probability and recur infinitely often. Continuous maps preserve the limit, so X_n^7\to0 and X_n^2+X_n+5\to5 almost surely. But let Y_n=n^2 with probability n^{-3} and 0 otherwise. Since \sum n^{-3}<\infty, Y_n\to0 almost surely, while E(Y_n^3)=n^3\to\infty.

Diagonalizability also appears as a side skill in this area of the paper. Eigenvectors belonging to distinct eigenvalues are linearly independent, so a square matrix with distinct eigenvalues is diagonalizable. Diagonalizability of A^2 does not transfer back to A. The 2\times2 matrix with a single 1 in position (1,2) is nilpotent, so its square is zero, which is diagonalizable, yet the matrix itself is not diagonalizable. A diagonalizable strictly upper triangular matrix has only the eigenvalue 0, so A=PDP^{-1} with D=0, and A must be the zero matrix. Finally, diagonalizable matrices need not be normal. The upper triangular matrix with rows (1,1) and (0,2) is diagonalizable, but AA^T\ne A^TA.

Central limit theorem

The central limit theorem says that for i.i.d. X_i with mean \mu and finite variance \sigma^2>0, the quantity \sqrt n(\bar X_n-\mu)/\sigma converges in distribution to N(0,1). The same reasoning applies to any i.i.d. terms W_i. Since \operatorname{Var}(\sum_i W_i)=n\operatorname{Var}(W), the sum divided by n has variance \operatorname{Var}(W)/n, and the correct scale is \sqrt n. Using the wrong scale gives either a degenerate limit at 0 or a divergent one.

Moments of a normal sample are needed when a question asks about a transformed sample. For X\sim N(\mu,\sigma^2), E(X^2)=\mu^2+\sigma^2 and E(X^4)=\mu^4+6\mu^2\sigma^2+3\sigma^4. With \mu=2 and \sigma^2=1, this gives E(X^2)=5, E(X^4)=43, and \operatorname{Var}(X^2)=18. For i.i.d. copies W_i of W=X^2, the quantities \operatorname{Var}(S_n)/n and E(S_n)/n equal \operatorname{Var}(W) and E(W) for every n, so expressions built from them have constant limits.

The delta method transfers a CLT through a differentiable function g. If \sqrt n(\bar Y_n-\mu)\to N(0,\tau^2), then \sqrt n(g(\bar Y_n)-g(\mu))\to N(0,g'(\mu)^2\tau^2). For a gamma population with shape \alpha and scale \beta, the mean is \alpha\beta and the variance is \alpha\beta^2. For g(x)=x(3-x) we have g'(x)=3-2x, so the limit variance at the mean \mu=\alpha\beta is (3-2\mu)^2\alpha\beta^2. Before reading off this variance, check that the centring constant is indeed g(\mu).

If X_i are i.i.d. Exp(1), then |X_1-X_2| is again Exp(1), by direct computation of its survival function. So the average of such absolute differences over disjoint pairs has mean and variance 1, and \sqrt n(\bar Y_n-1)\to N(0,1). With g(y)=e^{-y} and g'(1)=-e^{-1}, the delta method gives limit variance e^{-2}\approx0.135.

Convergence in second mean is a separate and often easier check. For a sample mean of i.i.d. variables with mean \mu and variance \sigma^2, E[((\bar Y_n-\mu)/\sqrt n)^2]=\sigma^2/n^2, which tends to 0. The \sqrt n-normalised quantity therefore converges to 0 in L^2, and hence in probability.

Slutsky's theorem combines the pieces. If A_n\to_d N and B_n\to_p c, then A_n+B_n\to_d N+c, and A_n/B_n\to_d N/c when c\ne0. For i.i.d. Poisson variables with mean \lambda, E(Y^2)=\lambda+\lambda^2, so the average of Y_i^2 converges almost surely to \lambda+\lambda^2 by the SLLN. Combining this with a standardised sum of mean-zero, unit-variance terms gives a normal limit. The ratio of the standardised sum to the average converges to a normal law with variance 1/(\lambda(1+\lambda))^2, and adding the average produces a normal limit with mean \lambda(1+\lambda) and variance 1.