Probability Distributions and Limit Theorems

Exponential distribution tail probability

The exponential distribution with rate \lambda > 0 has density f(x) = \lambda e^{-\lambda x} for x \ge 0, mean E[X] = 1/\lambda, and the closed-form tail P(X \ge x) = e^{-\lambda x} for x \ge 0. Because the tail has this simple form, one tail value determines the parameter. If P(X \ge a) = p, then e^{-\lambda a} = p, so \lambda = -\ln p / a. Do not confuse the rate with the mean. When a problem gives the mean \mu, the rate is \lambda = 1/\mu. For a continuous variable, P(X \ge x) = P(X > x), so the strictness of the inequality does not matter.

• tail probability — P(X \ge x) = e^{-\lambda x} for x \ge 0. Example: if \lambda = 0.5, then P(X \ge 2) = e^{-1}.
• rate versus mean — \lambda = 1/E[X]. Example: mean 4 means \lambda = 1/4.

Inversion example: if P(X \ge 4) = 1/8, then e^{-4\lambda} = 1/8, so \lambda = (\ln 8)/4 = (3\ln 2)/4.

The floor of an exponential variable is discrete and geometric-type. For integer k \ge 0, P(\lfloor X \rfloor = k) = P(k \le X < k+1) = e^{-\lambda k}(1 - e^{-\lambda}) = q^k(1-q) with q = e^{-\lambda}.

The memoryless property states P(X > s+t \mid X > s) = P(X > t) for s, t \ge 0. Therefore a conditional tail probability reduces to an unconditional one, whatever \lambda is. Example: if P(X > 5) = 0.6, then P(X > 10 \mid X > 5) = 0.6.

Central limit theorem approximation

The central limit theorem (CLT) says that for i.i.d. random variables with mean \mu and variance \sigma^2, the sum S_n is approximately N(n\mu, n\sigma^2) for large n. For a binomial variable Y \sim \text{Bin}(n,p), this gives Y \approx N(np, np(1-p)). A probability is then approximated by P(a \le Y \le b) \approx \Phi\left(\frac{b-\mu}{\sigma}\right) - \Phi\left(\frac{a-\mu}{\sigma}\right), where \Phi is the standard normal CDF. The standard deviation \sigma, not the variance, goes in the denominator.

Example: for Y \sim \text{Bin}(64, 1/2), the mean is 32 and the variance is 16, so P(28 \le Y \le 36) \approx \Phi(1) - \Phi(-1) \approx 0.683.

A continuity correction replaces a and b by a - 0.5 and b + 0.5 for an integer-valued variable. The CLT also applies to Poisson variables: a Poisson variable with mean n has variance n, so its standardized form is approximately standard normal for large n. Example: if N_n \sim \text{Poisson}(n), then P(N_n \le n) \approx \Phi\left(\frac{0.5}{\sqrt{n}}\right), which tends to \Phi(0) = 0.5 as n \to \infty. The partial sums \sum_{k=0}^{n} e^{-n} n^k / k! are exactly P(N_n \le n), so their limit is 0.5.

Poisson and normal distribution moments

A Poisson random variable with parameter \lambda > 0 has E[X] = \lambda and \operatorname{Var}(X) = \lambda. The mean and variance are equal, which is the feature that distinguishes it from the binomial, whose variance np(1-p) is smaller than its mean. Equal mean and variance is a standard test for whether Poisson is a plausible model.

A normal random variable N(\mu, \sigma^2) has mean \mu and variance \sigma^2. The standard normal is the special case \mu = 0, \sigma^2 = 1, so its mean is 0 and its variance is 1. Note that the second parameter is the variance, not the standard deviation.

• Poisson moments — E[X] = \operatorname{Var}(X) = \lambda. Example: a Poisson variable with \lambda = 3.5 has variance 3.5.
• standard normal moments — E[Z] = 0, \operatorname{Var}(Z) = 1.

These statements are true, so both the Poisson and standard normal facts are correct.

Expected waiting time for consecutive successes

For independent trials with success probability p, the number of trials until the first success is geometric, with expectation 1/p. The number of trials until two consecutive successes is different. Use first-step analysis. Let E_0 be the expected remaining number of trials when the last trial was not a success (or there is no previous trial), and E_1 when the last trial was a success. Then

E_0 = 1 + pE_1 + (1-p)E_0, E_1 = 1 + (1-p)E_0.

Solving gives E_0 = \frac{1+p}{p^2} = \frac{1}{p} + \frac{1}{p^2}. Do not confuse this with the expected number of trials for two successes that need not be consecutive, which is 2/p.

• consecutive successes — E = \frac{1}{p} + \frac{1}{p^2}. Example: with p = 1/3, E = 3 + 9 = 12.
• any two successes — E = 2/p, which is smaller than the consecutive case.

For a fair coin (p = 1/2), the expectation is 2 + 4 = 6.

Uniform distribution probabilities

A uniform random variable on [a,b] has density 1/(b-a) on that interval and zero elsewhere. For two independent uniform variables on rectangles, the joint density is the product of the marginal densities, so it is constant on the rectangle. An event's probability is then the area of the part of the rectangle where the event holds, multiplied by the constant density.

In general, P(\text{event}) = \dfrac{\text{area of event region within the rectangle}}{\text{area of the rectangle}}. Be careful to identify the region correctly, for example by requiring the ordering constraint and the range limits at the same time.

Example: if X \sim U[0,2] and Y \sim U[0,4] are independent, then P(X \le Y) is the area of \{(x,y): x \le y\} in the rectangle, which is \int_0^2 (4-x)\,dx = 6, divided by the rectangle area 8, giving 3/4.

Exponential distribution moments

For an exponential variable with rate \lambda, the mean and variance are E[X] = 1/\lambda and \operatorname{Var}(X) = 1/\lambda^2. The two are not equal in general, unlike the Poisson case. The variance is the square of the mean, so \operatorname{Var}(X) = (E[X])^2. The second moment is E[X^2] = 2/\lambda^2.