Hypothesis Testing and Nonparametric Methods

Neyman-Pearson test power

The Neyman–Pearson framework studies a simple null hypothesis H_0: \theta=\theta_0 against a simple alternative H_1: \theta=\theta_1. A test is of size \alpha if P_{\theta_0}(\text{reject }H_0)\le\alpha, and its power is \beta=P_{\theta_1}(\text{reject }H_0). Among all tests of size \alpha, the most powerful (MP) test has the largest power. The Neyman–Pearson lemma states that the MP test rejects H_0 when the likelihood ratio \Lambda(x)=f_1(x)/f_0(x) is large, so the critical region has the form \{x:\Lambda(x)>k\}, where k is chosen so that the size equals \alpha. Any strictly increasing function of \Lambda defines the same MP region, so the region can be written in terms of a more convenient statistic.

For a discrete model, a boundary point \Lambda=k may not allow the size to equal \alpha exactly. In that case the MP test is built by adding points in decreasing order of \Lambda until the size budget is used, and a randomized decision \phi(x)\in[0,1] is applied to the last point. The power is then E_{\theta_1}\phi(X), the f_1-weighted sum over the region. For example, let X take values 1,2,3,4 with f_0=(0.4,0.3,0.2,0.1) and f_1=(0.1,0.2,0.3,0.4). The ratios are 0.25,\,0.67,\,1.5,\,4. For \alpha=0.2, rejecting at x=4 gives size 0.1, and rejecting at x=3 with probability 1/2 raises the size to 0.2, so the power is 0.4+0.5\times0.3=0.55.

For a Bernoulli sample the likelihood ratio is an increasing function of the sum S, so the MP region is a set of large values of S, and both the size and the power are binomial tail probabilities. When the critical region is an interval \{x<c\} of a continuous variable, its size and power are simply F_{\theta_0}(c) and F_{\theta_1}(c), read directly from the distribution function. For the maximum X_{(n)} of n independent U(0,\theta) variables, P(X_{(n)}\le x)=(x/\theta)^n for 0<x<\theta, so the type-I error of the region \{X_{(n)}<c\} is (c/\theta_0)^n.

When f_0 and f_1 have disjoint supports, the likelihood ratio is infinite on the support of f_1, so an MP test rejects there with probability one and has power 1. Its size is governed only by its behaviour on the support of f_0, so the size may be strictly less than \alpha. Every MP test of level \alpha has E_{f_1}\phi(X)\ge\alpha, because the constant test \phi\equiv\alpha is itself of level \alpha. An MP test need not be unique: if a region has zero probability under f_1, altering the test there does not change its power, and the size constraint only has to be kept.

Two tests are equivalent in power if their power functions \gamma(\theta)=E_\theta\phi(X) coincide for all \theta, so candidate tests are compared by computing these functions over the parameter space. Here the sum of two independent exponential variables follows a gamma distribution, and the power is found by integrating the gamma density over the rejection set.

When the alternative distribution lies entirely inside the rejection region, the power equals one. For the family in which X is uniform on (\theta,\theta+1), the test that rejects for X>c has its largest null rejection probability at \theta=0, namely 1-c, so c=0.95 gives size 0.05, and every \theta>1 then gives power one.

Monotone likelihood ratio and UMP tests

A family \{f_\theta\} with a statistic T has a monotone likelihood ratio (MLR) in T when, for every \theta_1<\theta_2, the ratio f_{\theta_2}(x)/f_{\theta_1}(x) is a nondecreasing function of T(x). Exponential families in natural form have this property. For a zero-mean normal sample, the family indexed by the variance has MLR in \sum X_i^2. Under MLR, the one-sided problem H_0:\theta\le\theta_0 against H_1:\theta>\theta_0 has a uniformly most powerful (UMP) test of level \alpha, which rejects for large T. Its critical constant is fixed by the size at the boundary, because the power is monotone in \theta.

A UMP test is most powerful at every alternative in H_1 among tests of the same level. Since the constant test \phi\equiv\alpha has power \alpha everywhere, a UMP test has power at least \alpha at all alternatives, which means it is unbiased. The converse fails: an unbiased test with optimal properties need not be UMP. A UMP test may also have size strictly below \alpha in discrete models, and existence is not guaranteed for every problem, since two-sided problems usually lack UMP tests.

For a left-tailed alternative the direction reverses, and the test rejects for small values of T. A practical check is whether the rejection event is the one that becomes more likely as \theta moves into H_1. For a family whose maximum of two sample values is sufficient and has distribution e^{2(m-\theta)} below \theta, the UMP region is \{M\le c\}, and the power at a point is the probability of this event at that point.

In the discrete case, the p-value of a right-tailed test is the null probability of an outcome at least as extreme as the observed one, P_{\theta_0}(T\ge t_{\rm obs}), which is a binomial upper-tail sum when T is a count.

For a density of the form \beta e^{-x}(1-e^{-x})^{\beta-1}, the factor e^{-\sum x_i} does not involve \beta and cancels in the likelihood ratio. The remaining statistic \sum\ln(1-e^{-x_i}) is the one whose monotone likelihood ratio gives a UMP region of the form \{\sum\ln(1-e^{-x_i})>a\}.

Two-sided alternatives generally admit no UMP test, because the MP regions for \theta>\theta_0 and \theta<\theta_0 point in opposite directions. A UMP test can still exist in special configurations, such as when the support of the density depends on \theta so that both one-sided MP regions can be merged.

For a density proportional to x on (0,\theta), the distribution function is x^2/\theta^2, so the power of the region \{X>c\} at \theta is 1-c^2/\theta^2. This makes the power and size of such one-sided tests simple to calculate.

Likelihood ratio test

The likelihood ratio test (LRT) statistic is \lambda=\dfrac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)}, which takes values in [0,1]. Small values of \lambda are evidence against H_0. The statistic equals 1 when the maximum likelihood estimate lies in the null set, which is the case when the null value coincides with the MLE. The p-value is P_{H_0}(\lambda\le\lambda_{\rm obs}), so an observed \lambda=1 gives a p-value of 1.

In the multivariate normal model with unknown covariance, the LRT is equivalent to the Hotelling T^2 test. When the sample mean equals the hypothesized mean vector, the statistic is zero and the LRT statistic equals one, so the p-value is 1.

When the likelihood depends on the data only through a count, the LRT reduces to a rule on that count. If the agreement probability under H_1 exceeds that under H_0, the likelihood ratio increases in the count, so the test rejects for large counts. Both hypotheses are simple in this setting, so the LRT is also the Neyman–Pearson test.

For a normal sample with known variance, the LRT for H_0:\theta=0 against \theta\ne0 rejects when |\bar X| exceeds a critical value. Its power function is symmetric in \theta and never below the size, so the test is unbiased. No UMP test exists for the two-sided alternative. Symmetry matters for expectations: when \phi is symmetric under the sign change of the sample, the expectation of an odd function such as X_1\phi under \theta=0 is zero.

For two simple hypotheses, the likelihood ratio can be written in terms of |x_i| when the alternative density is a Laplace density. Completing the square in the exponent expresses the log-likelihood ratio as a decreasing function of \sum(|x_i|-1)^2, so the LRT rejects for large values of this sum.

Sign test for location parameter

The sign test for a location parameter tests H_0:\theta=\theta_0, where \theta is the median of a continuous distribution. Under H_0, each observation exceeds \theta_0 with probability 1/2 independently of the others, so the count S of observations above \theta_0 is \text{Bin}(n,1/2). This distribution does not depend on the underlying density f, which is the distribution-free property of the test. Under an alternative, the success probability \pi(\theta)=P_\theta(X>\theta_0) depends on both f and \theta, so the power function depends on \theta.

For a right-tailed sign test, the p-value is the upper-tail binomial probability P(S\ge s_{\rm obs})=\sum_{i=s_{\rm obs}}^{n}\binom{n}{i}(1/2)^n. The sum starts at the observed count, not at a fixed cut-off.

A critical region may also be written through order statistics. Since P(X_{(r)}>\theta_0) equals the probability that at most r-1 observations are at or below \theta_0, we have P(X_{(r)}>\theta_0)=\sum_{k=0}^{r-1}\binom{n}{k}2^{-n}, and the largest r with this value at most \alpha gives a level-\alpha test.

The Wilcoxon signed-rank statistic uses the magnitudes of the deviations as well as their signs. For a symmetric null distribution about \theta_0, its distribution does not depend on f and is symmetric about n(n+1)/4, so tail probabilities come from counting rank subsets with a given sum.

Kolmogorov-Smirnov test

The empirical distribution function F_n(x)=\frac{1}{n}\#\{i:X_i\le x\} estimates the true distribution function F. The Kolmogorov–Smirnov statistic D_n=\sup_x|F_n(x)-F_0(x)| is the largest vertical gap between F_n and the hypothesized F_0, and \sqrt n D_n is the test statistic. For continuous F_0, the distribution of \sqrt n D_n under H_0 is the same for every such F_0, because the probability integral transform maps the sample to i.i.d. uniform variables. By the Glivenko–Cantelli theorem, D_n\to0 almost surely as n\to\infty, so the statistic is consistent against fixed departures from H_0.