Probability Foundations and Random Variables

Bayes theorem posterior

Bayes' theorem reverses a conditional probability. For events A and B with P(B)>0, it states P(A\mid B)=\dfrac{P(B\mid A)\,P(A)}{P(B)}. Here P(A) is the prior, P(B\mid A) is the likelihood, P(B) is the evidence, and P(A\mid B) is the posterior, meaning the updated probability of A after B is observed.

The denominator is almost always computed with the law of total probability over a partition \{A_i\} of the sample space: P(B)=\sum_i P(B\mid A_i)P(A_i). When only an event T and its complement \bar T are involved, P(B)=P(B\mid T)P(T)+P(B\mid \bar T)P(\bar T). If a problem gives P(\bar T) instead of P(T), first use P(T)=1-P(\bar T).

A worked pattern: suppose one of two boxes is chosen with probabilities \tfrac{1}{4} and \tfrac{3}{4}, and a red ball is drawn with probabilities \tfrac{1}{2} and \tfrac{1}{4} respectively. Then P(\text{red})=\tfrac{1}{8}+\tfrac{3}{16}=\tfrac{5}{16}, so the posterior for the first box is P(\text{box 1}\mid \text{red})=\tfrac{1/8}{5/16}=\tfrac{2}{5}. The same structure applies when three or more boxes are chosen at random and the question asks which box a white ball most likely came from.

The most common mistake is confusing P(A\mid B) with P(B\mid A). In diagnostic testing, the test's accuracy is given as P(\text{positive}\mid\text{disease}), but the question usually asks for P(\text{disease}\mid\text{positive}). Because of the base-rate effect, a rare disease with an accurate test can still give a low posterior, since the false positives from the large healthy population can outnumber the true positives. For example, with prevalence 1\%, sensitivity 90\%, and false-positive rate 5\%, the posterior is \dfrac{0.009}{0.009+0.0495}\approx 0.15.

• prior — probability of a hypothesis before the evidence is seen.
• likelihood — probability of the observed evidence under a given hypothesis.
• posterior — probability of the hypothesis after the evidence is taken into account.

Event probability in coin tosses

For a finite sample space with equally likely outcomes, P(E)=\dfrac{|E|}{|\Omega|}. For n fair coins tossed independently, |\Omega|=2^n, and each outcome has probability 2^{-n}. Events defined by counts of heads or tails are then counted by combinations, such as \binom{n}{k} outcomes with exactly k heads.

Two events are mutually exclusive when A\cap B=\emptyset, which gives P(A\cap B)=0. Events defined by thresholds on counts can be incompatible when their requirements together exceed the number of tosses. For instance, with four coins, at least three heads and at least three tails cannot both occur.

If A\subseteq B, then P(A\cap B)=P(A). This monotonicity is often the quickest route to an intersection.

Covariance of indicator variables gives a related counting idea. If X=\mathbf{1}_A and Y=\mathbf{1}_B, then E[XY]=P(A\cap B), so \text{Cov}(X,Y)=P(A\cap B)-P(A)P(B). When A\subseteq B, this simplifies to \text{Cov}(X,Y)=P(A)\bigl(1-P(B)\bigr). Two fair coins give a concrete case in which X indicates two heads and Y indicates at least one head.

• mutually exclusive — two events that cannot occur together, so their intersection has probability zero.
• indicator variable — a random variable that equals 1 when an event occurs and 0 otherwise, with E[\mathbf{1}_A]=P(A).

Propositional tautologies

A compound proposition is a tautology if it is true under every truth assignment of its atomic propositions. It is a contradiction if it is false under every assignment, and a contingency if it is neither. Tautologies are checked with a truth table, but for implications there is a shortcut.

The implication p\Rightarrow q is false only when p is true and q is false. Therefore, to show that A\Rightarrow B is a tautology, list the assignments that make A true and check that B is true for each of them. If some assignment makes A true and B false, the implication is not a tautology.

Useful equivalences include p\Rightarrow q\equiv\neg p\vee q and p\Rightarrow q\equiv\neg q\Rightarrow\neg p, the contrapositive. For example, p\Rightarrow(p\vee q) is a tautology because whenever p is true, p\vee q is true, whereas (p\vee q)\Rightarrow p is not, because q true and p false is a counterexample.

• tautology — a formula that is always true.
• contrapositive — \neg q\Rightarrow\neg p, which is logically equivalent to p\Rightarrow q but not to its converse q\Rightarrow p.

Law of total expectation

The conditional expectation E[X\mid Y] is itself a random variable. It is a function of Y, defined by g(y)=E[X\mid Y=y], so it takes a value for each outcome of Y. Averaging this function over the distribution of Y returns the unconditional mean. This is the law of total expectation, or tower property: E\bigl[E[X\mid Y]\bigr]=E[X].

For discrete Y, the proof is E\bigl[E[X\mid Y]\bigr]=\sum_y E[X\mid Y=y]\,P(Y=y)=\sum_y\sum_x x\,P(X=x\mid Y=y)P(Y=y)=E[X]. The outer expectation removes the dependence on Y, so the result contains neither Y nor E[X\mid Y].

Example: let Y take the values 1 and 2 with probability \tfrac{1}{2} each, and let E[X\mid Y=1]=3 and E[X\mid Y=2]=7. Then E[X]=\tfrac{1}{2}(3)+\tfrac{1}{2}(7)=5. The same identity underlies the law of total variance, \text{Var}(X)=E[\text{Var}(X\mid Y)]+\text{Var}(E[X\mid Y]).

• tower property — the name for the identity E[E[X\mid Y]]=E[X] when conditioning is applied repeatedly.

CDF-based interval probability

The cumulative distribution function is F_X(x)=P(X\le x). The probability of an interval is obtained from it by P(a<X\le b)=F_X(b)-F_X(a). For a continuous random variable, P(X=a)=0 for every a, so the endpoints can be treated as open or closed without changing the probability: P(a\le X\le b)=F_X(b)-F_X(a^-)=F_X(b)-F_X(a).

Before evaluating F_X, the event must be rewritten as an interval on X. For c\ge 0, X^2\le c is equivalent to -\sqrt c\le X\le \sqrt c. The same method applies to a piecewise CDF: choose the piece that contains each endpoint, and evaluate the formula for that piece. For example, if F_X(x)=\tfrac{x}{4} on [0,4], then P(1\le X\le 3)=F_X(3)-F_X(1)=\tfrac{1}{2}.