Joint Distributions and Order Statistics

Order statistics of uniform distribution

For a random sample of size n from a continuous distribution with CDF F and density f, the kth order statistic X_{(k)} has density f_{(k)}(x)=\frac{n!}{(k-1)!\,(n-k)!}F(x)^{k-1}\{1-F(x)\}^{n-k}f(x). For the uniform distribution on [0,\theta] the CDF is x/\theta, so X_{(k)}/\theta follows a Beta distribution with parameters k and n-k+1, and E[X_{(k)}]=\frac{k\theta}{n+1}. To identify k from a given density of the form c\,x^{a}(L-x)^{b} on (0,L), read k-1=a and n-k=b, and then check that the constant c agrees with \frac{n!}{(k-1)!(n-k)!} divided by the scale factor. Because E[X_{(k)}]=k\theta/(n+1), the statistic \frac{n+1}{k}X_{(k)} is unbiased for \theta, which is the standard way to build unbiased estimators from order statistics. Conditioning an unbiased statistic on a complete sufficient statistic gives an estimator whose variance cannot exceed that of the original statistic, by the law of total variance: \mathrm{Var}(E[Y\mid\mathcal{G}])\le \mathrm{Var}(Y). For a uniform distribution on a symmetric interval (-\theta,\theta), the pair of extreme order statistics is sufficient but not complete, because the scale-invariant combination (X_{(1)}+X_{(n)})/(X_{(n)}-X_{(1)}) is ancillary with zero mean for every \theta. The range X_{(n)}-X_{(1)} is not ancillary, since it scales with \theta. For a uniform distribution on (a,\theta) with known a, the maximum Y has density n(y-a)^{n-1}/(\theta-a)^n, so E(Y)=a+\frac{n}{n+1}(\theta-a) and the estimator \frac{n+1}{n}(Y-a)+a is unbiased for \theta. For a uniform distribution on (-\theta/2,\theta/2), the likelihood is \theta^{-n} on the region \theta\ge 2\max_i|X_i|, so the MLE is 2\max_i|X_i|, not 2\max_i X_i, which can be negative. The statistic M=\max_i|X_i| is complete and sufficient with E(M)=\frac{n\theta}{2(n+1)}, so \frac{2(n+1)}{n}M is the UMVUE of \theta.

Conditional expectation from joint density

For a joint density f(x,y), the marginal density of X is f_X(x)=\int f(x,y)\,dy, and the conditional density of Y given X=x is f(y\mid x)=f(x,y)/f_X(x). The conditional expectation E(Y\mid X=x)=\int y\,f(y\mid x)\,dy is a function of x, and its graph is the regression curve of Y on X. A point (x_0,y_0) lies on this curve exactly when y_0=E(Y\mid X=x_0). For a regression of X on (Y,Z) with three variables, first integrate out the remaining variable to get the bivariate marginal, then compute E[X\mid Y=y,Z=z] as a ratio of two integrals in x. Solving E(X\mid Y=y_0)=c is an equation in y_0 after computing the conditional mean, which is linear in y for densities proportional to xy on a triangle. When the support depends on the variables, for instance y\ge x^2, the limits of integration depend on the conditioning value, and the joint density cannot be written as a product of marginals. Second moments follow from the tower property: E(Y^2)=E\{E(Y^2\mid X)\}, and for a uniform distribution on (0,b) the second moment is b^2/3. A necessary check before computing E(X+Y) is whether the expectations exist. If the marginal density behaves like c/x^{2} for large x, then \int |x| f_X(x)\,dx diverges, so E(X) and E(X+Y) are infinite even though the probabilities sum to one. The statement that E(Y\mid X=2) equals a certain value should be checked by writing the conditional density explicitly before integrating. Uncorrelated variables need not be independent: a symmetric joint support with E(XY)=0 can still have dependence through the support.

Poisson sum joint distribution

A joint probability mass function summed over one variable gives the marginal of the other. For example, if p(x,y)=e^{-2}\frac{1}{x!(y-x)!} for 0\le x\le y, then summing over x with the binomial identity \sum_{x}\binom{y}{x}=2^y gives P(Y=y)=e^{-2}2^y/y!, so Y is Poisson with mean 2. Given Y=y, the variable X is Binomial(y,1/2), so E(X\mid Y=y)=y/2 and E(X)=E(Y)/2. The joint moment generating function is obtained by conditioning: E(e^{uX+vY}\mid Y)=e^{vY}\left(\frac{1+e^u}{2}\right)^{Y}, and then the Poisson MGF \exp\{\lambda(e^t-1)\} is applied to the remaining factor. A useful pattern is a joint pmf of the form \frac{\lambda^n e^{-\lambda}}{2^n\,m!\,(n-m)!} with 0\le m\le n. Summing over m with the binomial identity gives a Poisson marginal with mean \lambda for Y. Summing over n with k=n-m gives a Poisson marginal for X with mean \lambda/2, because the factor e^{\lambda/2} from the sum over k combines with e^{-\lambda}. Dividing the joint pmf by the marginal of Y gives the conditional pmf \binom{n}{m}(1/2)^n, which is the binomial distribution \mathrm{Bin}(n,1/2). Given X=m, the excess Y-m is Poisson with mean \lambda/2, so E(Y\mid X=m)=m+\lambda/2.

Distribution of sample minimum

For a random sample X_1,\dots,X_n with CDF F, the event \min_i X_i\ge c means every observation is at least c, so P(\min_i X_i\ge c)=\{1-F(c)\}^n by independence. For the density f(x)=2x on (0,1), the CDF is F(x)=x^2, and the mean is E(X_1)=\int_0^1 x\cdot 2x\,dx=2/3. Combining these, P(\min_i X_i\ge E(X_1)) equals \{1-F(E(X_1))\}^n. For independent exponential variables U_k with rates \lambda_k, the minimum has survival function \prod_k e^{-\lambda_k y}=e^{-(\sum_k\lambda_k)y}, so the minimum is exponential with rate \sum_k\lambda_k. For an exponential variable Y with rate \lambda, E(Y)=1/\lambda and E(Y^2)=2/\lambda^2, so polynomial functions such as E(aY^2+bY) can be found by linearity. A common pitfall is to compute the minimum's distribution from the individual densities without taking the product of survival functions; the product is the correct route when the variables are independent but not identically distributed.