Probability and Statistics

Bayes' theorem

Bayes' theorem tells you how to revise the probability of a hypothesis after you observe some evidence. For events A and B with P(B)>0, P(A\mid B)=\dfrac{P(B\mid A)\,P(A)}{P(B)}. The term P(A) is the prior, P(B\mid A) is the likelihood, and P(A\mid B) is the posterior.

In exam problems the denominator is almost always expanded with the total probability theorem. If A_1,\dots,A_n form a partition of the sample space, then P(B)=\sum_{i} P(B\mid A_i)\,P(A_i). Therefore P(A_k\mid B)=\dfrac{P(B\mid A_k)P(A_k)}{\sum_i P(B\mid A_i)P(A_i)}.

When there are only two states, A and \bar A, the formula reads P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid \bar A)P(\bar A)}. This form suits reporter problems. A person who tells the truth with probability p reports the actual state with probability p and reports a wrong state with probability 1-p. The posterior probability of the actual state given a report must therefore combine the base rate of that state with p. It is not simply equal to p.

When the partition is by group, as when a population is split into subgroups with known shares and known rates of a trait, the posterior probability of a subgroup given the trait equals the share-weighted rate of that subgroup divided by the overall rate of the trait.

Key points and common confusions:

• P(A\mid B) versus P(B\mid A) — These are different quantities. The question usually asks for the first, while the data usually give the second. Swapping them is the classic error.
• prior versus posterior — The prior is the probability before the evidence; the posterior is after.
• base rate — A rare state with a small prior keeps a small posterior even when the evidence is fairly reliable. For example, if a condition has prevalence 0.01, a test with sensitivity 0.9 and false-positive rate 0.05 gives P(D\mid +)=\dfrac{0.9(0.01)}{0.9(0.01)+0.05(0.99)}=\dfrac{0.009}{0.0585}\approx 0.154, far below 0.9.
• Denominator — Always compute P(B) over all routes by which B can occur. Omitting one term gives a wrong answer.

Probability of union of events

For two events A and B, P(A\cup B)=P(A)+P(B)-P(A\cap B). The subtraction removes the overlap, which would otherwise be counted twice. For three events, P(A\cup B\cup C)=P(A)+P(B)+P(C)-P(A\cap B)-P(B\cap C)-P(A\cap C)+P(A\cap B\cap C). This is the inclusion-exclusion principle. The union means that at least one event occurs, so you add probabilities and subtract overlaps. You do not multiply them.

The value of P(A\cap B) depends on the relation between the events.

• Mutually exclusive — P(A\cap B)=0, so the union is just the sum.
• Independent — P(A\cap B)=P(A)P(B). Independence implies P(A\mid B)=P(A).
• Events with positive probabilities that are mutually exclusive can never be independent, since P(A\cap B)=0\neq P(A)P(B).

In network reliability problems, a route between two nodes works only if every edge on it is open, so the route event is an intersection of edge events. The event that at least one of two routes works is a union of those route events. When the two routes share no edge, their route events are independent, so each intersection becomes a product.