Bayes' Theorem

Thomas Bayes and the idea of reversing a conditional probability. Thomas Bayes (1702-1761) was an English statistician, philosopher, and Presbyterian minister, remembered for a specific and hugely influential result. Bayesian methods link a prior probability (belief held BEFORE running the experiment) and a conditional probability (the likelihood) to a posterior probability (belief held AFTER observing the outcome), via Bayes' rule. Bayesian probability treats probability more broadly, as a strength of belief or epistemic confidence given the available evidence -- not only as a long-run frequency.

Theorem 12.11 -- Bayes' Theorem. If $A_1,A_2,\ldots,A_n$ are mutually exclusive and exhaustive events (a partition of $S$) with $P(A_i)>0$ for every $i=1,2,\ldots,n$, and $B$ is any event with $P(B)>0$, then for each $i$:

Proof. By the Total Probability Theorem, $P(B)=\sum_j P(A_j)P(B/A_j)$ -- exactly the denominator above. By the Multiplication Theorem, $P(A_i\cap B)=P(B/A_i)\,P(A_i)$. By the definition of conditional probability, $P(A_i/B)=\dfrac{P(A_i\cap B)}{P(B)}$; substituting the two previous expressions for numerator and denominator gives the stated formula, linking $P(A_i/B)$ back to the (usually easier to state) $P(B/A_i)$.

Total Probability versus Bayes' Theorem -- two directions of the same setup. Given the same partition $A_1,\ldots,A_n$ and event $B$: Total Probability answers 'GIVEN each possible cause, what is the overall chance of the effect $B$?' -- summing forward from causes to effect. Bayes' Theorem answers the REVERSE question -- 'GIVEN that the effect $B$ was actually observed, what is the chance it came from cause $A_i$?' -- dividing 'this one cause's contribution to $B$' by 'every cause's total contribution to $B$' (which is exactly $P(B)$ from Total Probability). This is why Bayes' Theorem always needs the Total Probability computation as its denominator. …