Mathematics and Statistics in Ecology

Binomial probability

A binomial experiment has a fixed number n of independent trials, each with only two outcomes (success or failure) and a constant success probability p. The probability of exactly k successes is P(X=k)=\binom{n}{k}p^k(1-p)^{n-k}, where \binom{n}{k} counts the ways the successes can be placed among the trials. Binary records such as damaged or undamaged sampling units are modelled by the binomial distribution, because each unit is a single two-outcome trial with the same p. The number of males in a small group drawn from a population with a fixed sex ratio is also binomial, so the probability of exactly two males in a group of three is \binom{3}{2}p^2(1-p). A count of one of two types of items eaten from a fixed total is a discrete binomial count, not a continuous variable.

For 'at least one success' questions, use the complement: P(\text{at least one})=1-P(\text{none})=1-(1-p)^n. For example, with p=0.3 and n=2, the probability of at least one success is 1-0.7^2=0.51. When a fixed proportion of two types is sampled from a very large population, the draws behave approximately as independent trials, so the count of one type follows a binomial distribution with that proportion as p.

Do not confuse the binomial with the Poisson distribution, which counts events over a continuum of time or space with no fixed upper limit n, or with continuous distributions such as the normal, which cannot take only integer values.

Combinatorial probability

Many counting problems in ecology reduce to choosing a subset of distinct species, colours or genotypes in which the order of choice does not matter. The number of ways to choose k items from n distinct items is \binom{n}{k}=\frac{n!}{k!(n-k)!}, and \binom{n}{k}=\binom{n}{n-k}, which is a useful shortcut when k is more than half of n. For example, choosing 3 of 7 distinct colours gives \binom{7}{3}=35 possible sets.

When a grouping is formed from two independent pools, such as a set of one kind of species together with a set of another kind, the total number of compositions is the product of the two separate counts (multiplication principle). Each chosen pair from one pool combined with each chosen triple from the other pool gives a distinct composition.

For a diploid organism with two alleles at each of m independent loci, each locus has three genotypes (AA, Aa, aa), so the number of possible multilocus genotypes is 3^m.

A covering design asks for the minimum number of experimental units needed so that every required combination is tested. If each unit can test exactly one combination, the minimum number of units equals the total count of combinations that must be tested, which is the product of the numbers of subsets from each pool.

Statistical test selection

Choosing a test depends on the question, the type of data and the distributional assumptions. Comparing the means of three or more independent groups requires one-way analysis of variance, whereas the means of exactly two independent normal groups are compared with the Student's t-test.

The two-sample t statistic with pooled variance is t=\dfrac{\bar X_1-\bar X_2}{S_p\sqrt{1/n_1+1/n_2}}, where S_p is the pooled standard deviation. It has n_1+n_2-2 degrees of freedom and assumes normal populations with equal variances.

The association between two continuous variables is measured by Pearson's correlation coefficient r, which assumes approximately normal variables and measures linear association. Rank-based alternatives such as Spearman's correlation are used for non-normal or ordinal data.

Counts of individuals in categories compared with an expected Mendelian ratio are tested with the chi-square goodness-of-fit test. Chi-square is not used for continuous measurements or for comparing means.

Other tests have specific roles: Mann–Whitney U and Kruskal–Wallis are rank-based tests for two and several independent groups, Wilcoxon's signed-rank test is used for paired samples, the Shapiro–Wilk test checks normality, the Kolmogorov–Smirnov test compares distributions, and Kendall's coefficient of concordance measures agreement among rankings.

ANOVA F-test

In one-way analysis of variance, the F statistic is the ratio of the between-group mean square to the within-group (error) mean square. It is compared with an F distribution that has numerator and denominator degrees of freedom equal to the number of groups minus one and the total sample size minus the number of groups. With only two groups, this F statistic is equivalent to the square of the pooled t statistic.

The P-value is the probability, calculated under the null hypothesis of no difference, of obtaining a test statistic at least as extreme as the one observed. It is the upper-tail area of the appropriate distribution beyond the observed value. It is not the probability that the null hypothesis is true, it does not give the chance that repeated studies will show the same result, and it does not measure the size of an effect.

The F-test for equality of two variances is two-tailed. Under the null hypothesis the ratio of the variances is 1, and the observed ratio is compared with lower and upper critical values. If the statistic lies between the critical values, the null hypothesis is not rejected. Non-rejection does not prove that the null hypothesis is true, and the F-test says nothing about other distributional features such as skewness.

In linear regression, the F statistic is the mean square regression divided by the mean square error, where mean square regression equals the regression sum of squares divided by the number of predictors, and mean square error equals the residual sum of squares divided by the residual degrees of freedom. It tests the overall significance of the model, whereas the t statistic tests individual coefficients and R^2 measures the proportion of variation explained.

Linear regression assumptions

Ordinary least-squares regression estimates the intercept and slope by minimizing the sum of the squared residuals, which are the vertical differences between observed and fitted values. Squaring prevents positive and negative residuals from cancelling, so the plain sum of residuals is not a fitting criterion.

The main assumptions are a linear relationship between predictor and response, independence of observations, homoscedasticity (constant variance of the response across values of the predictor), and approximately normal residuals. Normality applies to the residuals, not necessarily to the raw response values. A fan-shaped scatter, in which the vertical spread grows or shrinks along the predictor axis, is heteroscedasticity and violates the equal-variance assumption.

Simple linear regression is the appropriate model when both the predictor and the response are continuous and the trend between them is roughly linear. Tests designed for group comparisons, such as the t-test and Kruskal–Wallis test, or for categorical association, such as the chi-squared test, do not model a continuous predictor.

Measures of variability and standard error

Measures of variability describe how far individual observations in a sample spread about their centre. The range is the difference between the maximum and minimum values, the inter-quartile range is the difference between the third and first quartiles, and the variance and standard deviation summarize the average squared or absolute deviation from the mean.

The standard error of the mean, SE=s/\sqrt{n}, is the standard deviation of the sampling distribution of the mean. It measures the precision of the sample mean as an estimate of the population mean, not the variability of individuals, and it shrinks as the sample size increases.

When two sample means are compared from their reported standard errors, the standard error of the difference is \sqrt{SE_1^2+SE_2^2}, and the t statistic is the difference divided by this value. A larger absolute t value gives a smaller p-value for comparable degrees of freedom, so the comparison depends on both the size of the mean difference and the precision of each mean.

Normal distribution and standard deviation

The normal distribution is symmetric and bell-shaped, with its centre at the mean \mu and its spread set by the standard deviation \sigma. Comparing two distributions, the one centred further to the right has the higher mean, and the one that spreads over a wider range has the higher standard deviation. These two properties are independent: a distribution may have a higher mean and a lower standard deviation than another.

For normal data, about 68.3% of observations lie within \mu\pm\sigma, about 95.4% within \mu\pm2\sigma, and about 99.7% within \mu\pm3\sigma. A 68% interval therefore corresponds to one standard deviation on either side of the mean.

Poisson distribution

The Poisson distribution gives the probability of k events, P(X=k)=\dfrac{e^{-\lambda}\lambda^k}{k!}, where \lambda is the mean number of events in a fixed interval or area. It is a discrete distribution on the nonnegative integers and is used for counts of independent events that occur at a constant average rate, such as captures per fixed time interval.

A key property is that the mean and the variance of a Poisson variable are both equal to \lambda. The index of dispersion, variance divided by mean, is therefore expected to be 1 for a random spatial distribution. Variance greater than the mean (overdispersion) indicates a clumped distribution, and variance less than the mean (underdispersion) indicates a regular, evenly spaced distribution.

Conditional probability of test results

The conditional probability of event A given event B is P(A\mid B)=\dfrac{P(A\cap B)}{P(B)}. In practice, conditioning restricts the sample space to the individuals who satisfy B, so the answer is the proportion of those individuals who also satisfy A.