Bayes' Theorem

Motivation: Reversing a Conditional Probability

The total probability theorem finds $P(A)$ given the conditional probabilities $P(A\mid E_i)$ of an effect $A$ given each possible cause $E_i$. Bayes' theorem answers the reverse question: having observed that $A$ has occurred, what is the probability that it was caused by a particular $E_i$? That is, it finds $P(E_i\mid A)$ from the $P(E_i)$ and $P(A\mid E_i)$.

Statement

Let $E_1,E_2,\dots,E_n$ be a partition of the sample space $S$ with $P(E_i)>0$ for every $i$, and let $A$ be any event with $P(A)>0$. Then, for each $i$,

Proof

By the definition of conditional probability and then the multiplication theorem (Section 2),

The denominator $P(A)$ is exactly the quantity given by the total probability theorem (Section 4), $P(A)=\sum_{j=1}^n P(E_j)P(A\mid E_j)$. Substituting this expression for $P(A)$ gives the stated formula. $\blacksquare$

Terminology

The events $E_1,\dots,E_n$ are called hypotheses; $P(E_i)$, known before any new evidence, is called the priori probability of $E_i$; and $P(E_i\mid A)$, recomputed after the evidence $A$ is observed, is called the posteriori probability of $E_i$ (given $A$). Bayes' theorem is precisely the rule for updating a priori belief into posteriori belief in light of new evidence.

Worked Illustration: A Diagnostic Test

Suppose $0.5\%$ of a population has a disease ($D$), so $P(D)=0.005$ and $P(D')=0.995$. A test correctly returns positive for $99\%$ of people who have the disease, $P(\text{Pos}\mid D)=0.99$, but also incorrectly returns positive for $2\%$ of healthy people, $P(\text{Pos}\mid D')=0.02$. Here $\{D,D'\}$ is a partition of the population, so Bayes' theorem gives

$P(D\mid \text{Pos}) = \frac{P(D)P(\text{Pos}\mid D)}{P(D)P(\text{Pos}\mid D)+P(D')P(\text{Pos}\mid D')} = \frac{(0.005)(0.99)}{(0.005)(0.99)+(0.995)(0.02)} = \frac{0.00495}{0.02485}\approx 0.199.$ …