Bayes' Theorem

Partition of a Sample Space

To understand Bayes' Theorem we first see how a sample space can be broken into useful pieces. Consider a random experiment with sample space $S$ and $n$ events $E_1, E_2, \ldots, E_n$ that are:

1. Mutually exclusive: $E_i \cap E_j = \emptyset$ for all $i \neq j$.
2. Exhaustive: $E_1 \cup E_2 \cup \cdots \cup E_n = S$.

Such a set $\{E_1, E_2, \ldots, E_n\}$ is called a partition of $S$.

For any event $A$, the partition lets us write $A$ as a union of disjoint pieces:

Since the $E_i$ are mutually exclusive, so are the $A \cap E_i$, hence:

Using $P(A \cap E_i) = P(E_i) \cdot P(A|E_i)$, we obtain the law of total probability:

This is the backbone of Bayes' Theorem.

---

The Problem of Reverse Probability

Consider the opening example. Bag I has 2 white and 3 red balls; Bag II has 4 white and 5 red. A bag is chosen at random, then a ball is drawn. It is easy to find the probability of drawing white given which bag was chosen. But the real question is the reverse: if the drawn ball is white, what is the probability it came from Bag II?

This is a reverse probability problem — we know the effect (the ball's colour) and want to infer the cause (which bag). Bayes' formula uses conditional probability to reverse the conditioning, from $P(\text{effect}|\text{cause})$ to $P(\text{cause}|\text{effect})$.

---

Statement of Bayes' Theorem

Let $E_1, E_2, \ldots, E_n$ be a partition of $S$ with $P(E_i) > 0$, and let $A$ be any event with $P(A) > 0$. Then for any $k = 1, 2, \ldots, n$:

The denominator is just $P(A)$ by the law of total probability.

---

Proof of Bayes' Theorem

›Proof

By the definition of conditional probability, $P(E_k|A) = \dfrac{P(E_k \cap A)}{P(A)}$.

The multiplication rule gives $P(E_k \cap A) = P(E_k) \cdot P(A|E_k)$, and the law of total probability gives $P(A) = \sum_{i=1}^{n} P(E_i) P(A|E_i)$. Substituting:

$P(E_k|A) = \frac{P(E_k) P(A|E_k)}{\displaystyle\sum_{i=1}^{n} P(E_i) P(A|E_i)}$

---

Understanding the Terms

• $P(E_k)$ — the prior probability: our initial belief in $E_k$ before observing data.
• $P(A|E_k)$ — the likelihood: the probability of observing $A$ given $E_k$.
• $P(E_k|A)$ — the posterior probability: our updated belief in $E_k$ after observing $A$.
• $P(A)$ — the evidence: the total probability of $A$ under all causes.

---

Worked Example: The Two-Bag Problem

Step 1 — Define events. Let $E_1$ = Bag I chosen, $E_2$ = Bag II chosen, $A$ = a white ball is drawn.

Step 2 — Given probabilities. A bag is chosen at random, so $P(E_1) = P(E_2) = \frac{1}{2}$. Bag I has 2 white of 5, Bag II has 4 white of 9:

Step 3 — Total probability of white.

Step 4 — Apply Bayes' Theorem for $P(E_2|A)$:

---

Related Results

Property (I): Bayes' Rule for Two Events

When the partition is just $E$ and its complement $E'$, Bayes' Theorem becomes:

The denominator is the two-event law of total probability $P(A) = P(E) P(A|E) + P(E') P(A|E')$.

Property (II): Posterior Odds Ratio

For any two events $E_i$, $E_j$ from the partition:

so the posterior odds equal the prior odds times the likelihood ratio.

›Proof

Dividing $P(E_i|A) = \frac{P(E_i) P(A|E_i)}{P(A)}$ by $P(E_j|A) = \frac{P(E_j) P(A|E_j)}{P(A)}$, the common $P(A)$ cancels, giving the result. …