3.1 Bayes’ Law
Bayes’ Law is a clever re-ordering of the relationship between joint probability and conditional probability, \(P(\theta D) = P(\theta|D)P(D) = P(D|\theta)P(\theta)\), into
\[P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)}.\]
\(P(\theta)\) is the strength of your belief in \(\theta\) prior to considering the data \(D\).
\(P(D|\theta)\) is the likelihood of observing \(D\) from a generative model with parameter \(\theta\). Note that likelihood is the probability density, and is not quite the same as probability. For a continuous variable, likelihoods can sum to greater than 1. E.g.,
dbinom(seq(1, 100, 1), 100, .5)sums to 1, butdnorm(seq(0,50,.001), 10, 10)sums to 841.\(P(D)\) is the likelihood of observing \(D\) from any prior. It is the marginal distribution, or prior predictive distribution of \(D\). The likelihood divided by the marginal distribution is the proportional adjustment made to the prior in light of the data.
\(P(\theta|D)\) is the strength of your belief in \(\theta\) posterior to considering \(D\).
Bayes’ Law is useful for evaluating medical tests. A test’s sensitivity is its probability of yielding a positive result \(D\) when condition \(\theta\) exists. \(P(D|\theta)\) is a test sensitivity, \(\mathrm{sens}\). \(P(\theta)\) is the probability of \(\theta\) prior to the test (e.g., the general rate in society), \(\mathrm{prior}\). The numerator of Bayes’ Law, the joint probability \(P(D \theta) = P(D|\theta)P(\theta)\), is \(\mathrm{sens\times prior}\). A test’s specificity is the probability of observing negative test result \(\hat{D}\) when the condition does not exist, \(\hat{\theta}\). The specificity is the compliment of a false positive test result, \(P(\hat{D} | \hat{\theta}) = 1 - P(D | \hat{\theta})\). The denominator of Bayes’ Law is the overall probability of a positive test result, \(P(D) = P(D|\theta)P(\theta) + P(D|\hat\theta)P(\hat\theta)\) or in terms of sensitivity and specificity, \(P(D) = \mathrm{(sens \times prior) + (1 - spec)(1 - prior)}\).
Example. Suppose E. Coli is typically present in 4.5% of samples, and an E. Coli screen has a sensitivity of 0.95 and a specificity of 0.99. Given a positive test result, what is the probability that E. Coli is actually present?
\[P(\theta|D) = \frac{.95\cdot .045}{.95\cdot .045 + (1 - .99)(1 - .045)} = \frac{.04275}{.04275 + .00955} = \frac{.04275}{.05230} = 81.7\%.\]
The elements of Bayes’ Law come directly from the contingency table. The first row is the positive test result. The probability of E. Coli is the joint probability of E. coli and a positive test divided by the probability of a positive test
| E. Coli | Safe | Total | |
|---|---|---|---|
| + Test | .95 * .045 = 0.04275 | .01 * .955 = 0.00955 | 0.05230 |
| - Test | .05 * .045 = 0.00225 | .99 * .955 = 0.94545 | 0.94770 |
| Total | 0.04500 | 0.95500 | 1.00000 |