Probability and Statistical Inference

Continuous probability density functions

A continuous random variable X is described by a probability density function f(x). Probabilities are areas under the curve, not sums of point masses. For f to be a valid density, two conditions must hold: f(x)\ge 0 for all x, and \int_{-\infty}^{\infty} f(x)\,dx=1. The second condition is used to find an unknown constant in a density. Integrate the expression over its support, set the result equal to 1, and solve. For a density proportional to x^{-1/2} on (0,a), the constant is c=1/(2\sqrt{a}), because \int_0^a c\,x^{-1/2}\,dx=2c\sqrt{a}. For example, if f(x)=kx on 0<x<2, then \int_0^2 kx\,dx=2k=1, so k=\tfrac12. Polynomial densities work the same way. For f(x)=kx+\tfrac12 on [0,1], the normalisation gives \tfrac k2+\tfrac12=1, so k=1.

The median m of a continuous variable satisfies F(m)=P(X\le m)=\tfrac12, where F(x)=\int_{-\infty}^{x} f(t)\,dt is the cumulative distribution function. To find it, compute F on the support and then solve F(m)=\tfrac12. For f(x)=2/x^3 on x>1, F(x)=1-1/x^2, so 1-1/m^2=\tfrac12 gives m=\sqrt{2}.

Estimator bias and consistency

An estimator \hat\theta of a parameter \theta is unbiased if E[\hat\theta]=\theta. Its bias is E[\hat\theta]-\theta. Consistency is a large-sample property: \hat\theta is consistent if P(|\hat\theta-\theta|>\varepsilon)\to 0 for every \varepsilon>0 as n\to\infty. The two properties are independent of each other. An estimator may be biased for every finite n and still be consistent, and an unbiased estimator may fail to be consistent. The variance estimator that divides by n is a standard case of a biased but consistent estimator, since its bias vanishes as the sample grows.

The same vocabulary of bounds also appears in probability questions. For any two events A and B, the intersection is constrained by \max(0,\,P(A)+P(B)-1)\le P(A\cap B)\le \min(P(A),P(B)), and both bounds can be attained. For example, with P(A)=0.6 and P(B)=0.3, P(A\cap B) lies between 0 and 0.3.

Expected value of order statistics

For a random sample X_1,\dots,X_n, the maximum Z=\max(X_1,\dots,X_n) is an order statistic. Its distribution follows from P(Z\le z)=\big(P(X\le z)\big)^n, because every observation must fall at or below z. When the population is discrete, the expectation is found by listing the possible values of Z and their probabilities, and then summing z\,P(Z=z). For example, take two independent draws, each equally likely to be 0 or 2. Then P(Z=0)=(1/2)^2=1/4, so P(Z=2)=3/4, and E[Z]=2\cdot\tfrac34=\tfrac32.

Continuous uniform distribution

A continuous uniform distribution on (a,b) has density f(x)=\dfrac{1}{b-a} for a<x<b and zero elsewhere. Every subinterval of equal length therefore has equal probability. Its mean is \dfrac{a+b}{2}, and its variance is \dfrac{(b-a)^2}{12}. This follows from E[X^2]=\dfrac{a^2+ab+b^2}{3}. When the support is symmetric, (-a,a), the mean is zero and the variance is a^2/3. A common mistake is to use the half-length a in the variance formula in place of the length b-a. For example, if X\sim U(2,8), then \operatorname{Var}(X)=\dfrac{36}{12}=3.

Maximum likelihood estimation of normal parameters

Maximum likelihood estimation chooses the parameter values that maximise the likelihood L(\theta)=\prod_{i=1}^n f(x_i;\theta), or equivalently the log-likelihood. For a normal sample N(\mu,\sigma^2) with both parameters unknown, the condition for \mu gives \hat\mu=\bar x. Substituting this into the condition for \sigma^2 gives \hat\sigma^2_{MLE}=\dfrac{1}{n}\sum_{i=1}^n (x_i-\bar x)^2. The divisor is n, not n-1. Since E[\hat\sigma^2_{MLE}]=\dfrac{n-1}{n}\sigma^2, the MLE is biased downward. Its bias -\sigma^2/n disappears as n\to\infty, so the estimator is consistent. Multiplying by \dfrac{n}{n-1} gives the unbiased sample variance, which is not the MLE. For n=5, the expectation of the MLE is \tfrac45\sigma^2.

Discrete uniform distribution moments

A discrete uniform distribution assigns the same probability 1/k to each of k distinct values. Probabilities are computed by counting the favourable values and dividing by k. For the values 1,2,\dots,n, the mean is \dfrac{n+1}{2} and the variance is \dfrac{n^2-1}{12}. The variance is computed as \operatorname{Var}(X)=E[X^2]-(E[X])^2, where E[X^2]=\sum x_i^2\,p(x_i). The mean must be found first and squared before it is subtracted. For example, if X is uniform on \{2,4,6\}, then E[X]=4, E[X^2]=\tfrac{56}{3}, and \operatorname{Var}(X)=\tfrac83.

Conditional expectation from joint density

The conditional density of X given Y=y is f_{X\mid Y}(x\mid y)=\dfrac{f(x,y)}{f_Y(y)}, where f_Y(y)=\int f(x,y)\,dx is the marginal density of Y. The conditional expectation is E[X\mid Y=y]=\int x\,f_{X\mid Y}(x\mid y)\,dx, a function of y. Two steps are needed. First find the marginal of the conditioning variable, because the conditional density must be normalised by it rather than by the constant in the joint density. Then evaluate at the given value of y before integrating. For example, if f(x,y)=x+y on the unit square, then f_Y(y)=\tfrac12+y and E[X\mid Y=y]=\dfrac{1/3+y/2}{1/2+y}. At y=\tfrac12 this equals \tfrac{7}{12}.