Bayes' Theorem

Bayes' theorem (also called Bayes' law or Bayes' rule) is a direct extension of conditional probability that lets us go 'in reverse' - from knowing an outcome $A$ occurred, back to the probability that a particular underlying cause $E_i$ produced it. It is especially useful for computing such posterior probabilities.

Theorem. If $E_1,E_2,\ldots,E_n$ are mutually exclusive and exhaustive events with $P(E_i)\ne0$ for each $i$, and $A$ is any event (a subset of the union of the $E_i$) with $P(A)>0$, then $P(E_i/A)=\frac{P(E_i)\cdot P(A/E_i)}{\displaystyle\sum_{i=1}^{n}P(E_i)\cdot P(A/E_i)}.$

Proof. Because the $E_i$ are exhaustive, $A$ can be written as $A=(A\cap E_1)\cup(A\cap E_2)\cup\cdots\cup(A\cap E_n)$, and because the $E_i$ are mutually exclusive, so are the pieces $A\cap E_i$. Property 10 (additivity over mutually exclusive events) then gives $P(A)=\sum_{i=1}^{n}P(A\cap E_i)$ - this is the law of total probability, used repeatedly to compute $P(A)$ in every worked example below. By the multiplication theorem, each term is $P(A\cap E_i)=P(E_i)\cdot P(A/E_i)$, so $P(A)=\sum_i P(E_i)P(A/E_i)$. Meanwhile $P(E_i/A)=P(A\cap E_i)/P(A)=P(E_i)P(A/E_i)/P(A)$; substituting the sum for $P(A)$ in the denominator gives the theorem.

Three kinds of probability appear in the formula: the $P(E_i)$ are prior probabilities, known before the experiment; the $P(A/E_i)$ are likelihood probabilities, telling us how likely $A$ is under each possible cause $E_i$; and the $P(E_i/A)$ are posterior probabilities, obtained only after the experiment (after $A$ is observed). …