Random Variables and Moments

Moments of a random variable

The r-th raw moment of a random variable X is E(X^r), and the r-th central moment is E[(X-\mu)^r], where \mu=E(X). For a discrete variable, E(X^r)=\sum_x x^r p(x). For a continuous variable, E(X^r)=\int x^r f(x)\,dx, provided the integral converges absolutely. The variance is \mathrm{Var}(X)=E(X^2)-[E(X)]^2, which is the identity used most often in these items. To find E[g(X)] for a polynomial g, expand g and apply linearity, so that only a few raw moments are needed. For the normal distribution, the raw moments follow from the mean and variance, and a useful check is Stein's identity E[(X-\mu)h(X)]=\sigma^2E[h'(X)].

For a non-negative integer-valued variable, the tail-sum formula gives E(X)=\sum_{n\ge 1}P(X\ge n). Since X^2=\sum_{n=1}^{X}(2n-1), one also gets E(X^2)=\sum_{n\ge 1}(2n-1)P(X\ge n). Both formulas use only the distribution function, so they avoid computing the pmf directly.

Many moment questions start from a counting model. When a variable counts how many items of each type appear among independent draws, its joint pmf is a multinomial probability with coefficient n!/(k_1!\,k_2!\,k_3!) times the product of the category probabilities. A variable defined as the minimum number of trials needed to reach a stopping condition has a pmf obtained by counting equally likely orderings of the outcomes, and its moments are then computed from that pmf.

Moment generating function

The moment generating function (MGF) of X is M_X(t)=E(e^{tX}), defined for t in an open interval around 0. Differentiating r times and setting t=0 gives M_X^{(r)}(0)=E(X^r). Hence the fourth derivative at zero is the fourth raw moment. For a variable with finitely many values x_j and probabilities p_j, M_X(t)=\sum_j p_je^{tx_j}, and the fourth derivative at 0 equals \sum_j p_jx_j^4.

Two properties are used repeatedly. First, if X and Y are independent, M_{X+Y}(t)=M_X(t)M_Y(t). Second, the MGF determines the distribution uniquely, so a recognised form identifies it. For example, e^{\lambda(e^t-1)} is the MGF of a Poisson distribution, (q+pe^t)^n is that of a binomial distribution, and (1-2t)^{-k/2} is that of a chi-square distribution with k degrees of freedom. A product of two recognised MGFs, such as a uniform and an exponential, identifies the variable as a sum of independent variables, and its probabilities can then be computed by integration.

A weighted sum of MGFs is a mixture. If H(t)=pM_1(t)+(1-p)M_2(t), then the variable equals one with a distribution having MGF M_1 with probability p and otherwise has MGF M_2. A factor e^{at} in an MGF shifts the variable by a. When the joint MGF M_{X,Y}(u,v) factorises as g(u)h(v) on a neighbourhood of the origin, X and Y are independent, and expectations such as E(X^2/Y) can then be split into products. Conversely, if a marginal MGF is known and the sum's MGF is given, the MGF of the other summand is the ratio of the two MGFs. If an MGF equals e^{ct}, the variable is degenerate at c and its variance is 0.

CDF of mixed random variable

The distribution function F(x)=P(X\le x) is non-decreasing and right-continuous, with limits 0 at -\infty and 1 at +\infty. The jump of F at a point a gives the point mass, P(X=a)=F(a)-F(a^-). A mixed variable has a CDF with both jump parts and continuous increasing parts. Interval probabilities then need care at the endpoints: P(a<X\le b)=F(b)-F(a), P(a\le X\le b)=F(b)-F(a^-), and P(a<X<b)=F(b^-)-F(a).

Because F is right-continuous, the value P(X\le a)=F(a) includes any atom at a, even when F jumps there. A median m is any value with F(m)\ge 1/2 and F(m^-)\le 1/2, and when F is continuous and strictly increasing at m, the median is unique.

Probabilities of shrinking intervals do not always tend to zero. If F has an atom at a, then P(a-1/n<X<a+1/n) tends to P(X=a) as n\to\infty, while the probability of a one-sided interval whose endpoint is a is governed by the continuous parts of F. Such limits are evaluated by splitting the interval at the atoms.

The conditions that define a CDF, such as the value at a point or continuity of a piece, often determine unknown constants. For instance, matching F to a known value at a jump or to a prescribed median fixes the constant.

Conditional expectation of discrete variables

For an event B with P(B)>0, the conditional expectation is E(X\mid B)=\sum_x xP(X=x\mid B)=\frac{E(X\mathbf{1}_B)}{P(B)}. When all outcomes are equally likely, E(X\mid B) is the arithmetic mean of the values of X over the outcomes that belong to B. Listing the outcomes in B and averaging is often the fastest route.

When a random index Y decides the experiment, the total expectation formula applies: E(Z)=\sum_k P(Y=k)E(Z\mid Y=k). Given Y=k, the maximum of k independent variables is at most a level c with probability F(c)^k, and averaging these over the pmf of Y gives the answer.

When the complement of an event is simpler, compute the sum of all values, subtract the sum over the excluded values, and divide by the number of remaining values. For example, for a uniform variable on a set of consecutive odd integers, the sum of the first n odd numbers is n^2, so excluded multiples are removed by subtraction.

For independent identically distributed geometric variables on \{0,1,2,\dots\} with parameter p, the conditional distribution of X given X+Y=s is uniform on \{0,1,\dots,s\}, so E(X\mid X+Y)=(X+Y)/2. Also P(X=Y)=\sum_x p^2(1-p)^{2x}=p/(2-p).

Distribution of transformed random variable

If Y=g(X), find the distribution of Y by writing P(Y\le y) as the probability of the set \{x: g(x)\le y\}, and then use the CDF of X. For a non-monotone transformation such as a square, the set typically splits into two intervals, and both must be counted. The event Y\le y with Y=1-X^2, for example, becomes a condition on |X|.

For a strictly monotone g, the density is f_Y(y)=f_X(g^{-1}(y))\left|\frac{d}{dy}g^{-1}(y)\right|. For a positive variable X with a Gamma density and Y=\ln X, the moments of \ln X come from the derivative of \ln E(X^s) at s=0. For a Gamma variable with shape \alpha and rate \beta, this gives E(\ln X)=\psi(\alpha)-\ln\beta, where \psi is the digamma function.

Jensen's inequality states that for a concave function \varphi, E[\varphi(X)]\le\varphi(E(X)). Since \ln is concave, E(\ln X)\le\ln E(X) for a positive variable with finite mean, with strict inequality when X is non-degenerate.