16 Introduction to the Bayes Factor and Posterior Odds: The Competition of Ideas

The Bayes factor is a formula that tests the plausibility of one hypothesis by comparing it to another. The result tells us how many times more likely one hypothesis is than the other.

16.1 Revisiting Bayes’ Theorem

\[ P(H \mid D) = \frac{{P(H)} \times P(D \mid H) }{P(D)} \tag{16.1}\]

$P(H \mid D)$: Posterior Probability, which tells us how strongly we should believe in our hypothesis, given our data.
$P(H)$: Prior Probability or Prior Belief, the probability of our hypothesis prior to looking at the data.
$P(D \mid H)$: Likelihood of getting the existing data if our hypothesis were true.
$P(D)$ is the probability of the data observed independent of the hypothesis. We need P(D) in order to make sure that our posterior probability is correctly placed somewhere between 0 and 1.

$P(D)$ is … totally unnecessary if all we care about is comparing the relative strength of two different hypotheses. … For these reasons, we often use the proportional form of Bayes’ theorem.

\[P(H \mid D) \propto P(H) \times P(D \mid H) \tag{16.2}\]

the posterior probability of our hypothesis is proportional to the prior multiplied by the likelihood. We can use this to compare two hypotheses by examining the ratio of the prior belief multiplied by the likelihood for each hypothesis using the ratio of posteriors formula:

\[\frac{P(H_{1}) \times P(D \mid H_{1})}{P(H_{2}) \times P(D \mid H_{2})} \tag{16.3}\]

if the ratio is 2, then $H_{1}$ explains the observed data twice as well as $H_{2}$, and if the ratio is $\frac{1}{2}$, then $H_{2}$ explains the data twice as well as $H_{1}$.

16.2 Building a Hypothesis Test Using the Ratio of Posteriors

The ratio of posteriors formula gives us the posterior odds, which allows us to test hypotheses or beliefs we have about data.

To better understand the posterior odds, we’ll break down the ratio of posteriors formula into two parts: the likelihood ratio, or the Bayes factor, and the ratio of prior probabilities.

16.2.1 The Bayes Factor

\[\frac{P(D \mid H_{1})}{P(D \mid H_{2})} \tag{16.4}\]

What this ratio tells us is the likelihood of what we’ve seen given what we believe to be true compared to what someone else believes to be true.

The key here is that in Bayesian reasoning, we don’t worry about supporting our beliefs—we are focused on how well our beliefs support the data we observe. In the end, data can either confirm our ideas or lead us to change our minds.

16.2.2 Prior Odds

So far we have assumed that the prior probability of each hypothesis is the same. This is clearly not always the case: a hypothesis may explain the data well even if it is very unlikely.

\[\frac{P(H_{1})}{P(H_{2})}\]

This ratio compares the probability of two hypotheses before we look at the data. When used in relation to the Bayes factor, this ratio is called the prior odds in our $H_{1}$ and written as $O(H_{1})$. This representation is helpful because it lets us easily note how strongly (or weakly) we believe in the hypothesis we’re testing. When this number is greater than 1, it means the prior odds favor our hypothesis, and when it is a fraction less than 1, it means they’re against our hypothesis. For example, $O(H_{1}) = 100$$ means that, without any other information, we believe $H_{1}$ is 100 times more likely than the alternative hypothesis.

16.2.3 Posterior Odds

\[\text{posterior odds} = O(H_{1})\frac{P(D \mid H_{1})}{P(D \mid H_{2})} \tag{16.5}\]

Table 16.1: Guidelines for Evaluating Posterior Odds
Posterior odds	Strength of evidence
1 to 3	Interesting, but nothing conclusive
3 to 20	Looks like we’re on to something
20 to 150	Strong evidence in favor of $H_{1}$
> 150	Overwhelming evidence

16.2.4 Empty: Guidelines for Evaluating Posterior Odds

16.2.5 Empty: Self-Diagnosing Rare Diseases Online

16.3 Wrapping Up

In this chapter, you learned how to use the Bayes factor and posterior odds to compare two hypotheses.

16.4 Exercises

Try answering the following questions to see how well you understand the Bayes factor and posterior odds. The solutions can be found at https://nostarch.com/learnbayes/.

16.4.1 Exercise 16-1

Returning to the dice problem, assume that your friend made a mistake and suddenly realized that there were, in fact, two loaded dice and only one fair die. How does this change the prior, and therefore the posterior odds, for our problem? Are you more willing to believe that the die being rolled is the loaded die?

Let’s recapitulate:

Suppose your friend has a bag with three six-sided dice in it, and one die is weighted so that it lands on 6 half the time. The other two are traditional dice whose probability of rolling a 6 is ⅙. Your friend pulls out a die and rolls 10 times, with the following results: $6, 1, 3, 6, 4, 5, 6, 1, 2, 6$

Definition 16.1 (Bayes Factor) \[\frac{P(D \mid H_{1})}{P(D \mid H_{2})}\]

It turned out that $H_{1}$ — the loaded dice — explain the data 3.77 times better than the fair dice with $H_{2}$.

However, this is true only if $H_{1}$ and $H_{2}$ are both just as likely to be true in the first place.

But we know now for this exercise that there are two loaded dice in the bag and only one fair die, which means that each hypothesis was not equally likely. Based on the distribution of the dice in the bag, we know that these are the prior probabilities for each hypothesis:

\[P(H_{1} = \frac{2}{3}); P(H_{2} = \frac{1}{3})\] > From these, we can calculate the prior odds for $H_{1}$:

\[\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{\frac{2}{3}}{\frac{1}{3}} = \frac{6}{3} = 2\]

With our prior odds for H1, we can now compute our full posterior odds:

\[\text{posterior odds} = O(H_{1}) = \frac{P(D \mid H_{1})}{P(D \mid H_{2})} = 2 \times 3.77 = 7.54\]

Solution 16.1 Yes, I am now more willing to believe that the die being rolled is the loaded die!

16.4.2 Exercise 16-2

Returning to the rare diseases example, suppose you go to the doctor, and after having your ears cleaned you notice that your symptoms persist. Even worse, you have a new symptom: vertigo. The doctor proposes another possible explanation, labyrinthitis, which is a viral infection of the inner ear in which 98 percent of cases involve vertigo. However, hearing loss and tinnitus are less common in this disease; hearing loss occurs only 30 percent of the time, and tinnitus occurs only 28 percent of the time. Vertigo is also a possible symptom of vestibular schwannoma, but occurs in only 49 percent of cases. In the general population, 35 people per million contract labyrinthitis annually. What is the posterior odds when you compare the hypothesis that you have labyrinthitis against the hypothesis that you have vestibular schwannoma?

Labyrinthitis:

\[P(D \mid H_{1}) = 0.98 (\text{vertigo}) \times 0.30 (\text{hearing loss}) \times 0.28 (\text{tinnitus}) = 0.082\]

Vestibular schwannoma

\[P(D \mid H_{2}) = 0.49 (\text{vertigo}) \times 0.94 (\text{hearing loss}) \times 0.89 (\text{tinnitus}) = 0.41\]

Bayes factor:

\[\frac{P(D \mid H_{1})}{P(D \mid H_{2})} = \frac{0.08}{0.48} = 0.17\]

Including prior odds:

\[\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{\frac{35}{1,000,000}}{\frac{11}{1,000,000}} = \frac{35}{11} = 3.18\] Based on prior information alone, a given person is about only about 3 times more likely to have labyrinthitis than vestibular schwannoma. Now let’s compute the full posterior odds to see if the situation gets better:

\[\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{35}{11} \times 6 = 19,09\]

Solution 16.2 The probability of having labyrinthitis is 19 times higher than vestibular schwannoma.

Warning

Will Kurt has another result because he is calculating the labyrinthitis against the earwax impaction!

--- engine: knitr --- # Introduction to the Bayes Factor and Posterior Odds: The Competition of Ideas > The `r glossary("Bayes factor")` is a formula that tests the plausibility of one hypothesis by comparing it to another. The result tells us how many times more likely one hypothesis is than the other. ## Revisiting Bayes’ Theorem $$ P(H \mid D) = \frac{{P(H)} \times P(D \mid H) }{P(D)} $$ {#eq-bayes-theorem} - $P(H \mid D)$: `r glossary("Posterior Probability")`, which tells us how strongly we should believe in our hypothesis, given our data. - $P(H)$: `r glossary("Prior Probability")` or Prior Belief, the probability of our hypothesis prior to looking at the data. - $P(D \mid H)$: `r glossary("Likelihood")` of getting the existing data if our hypothesis were true. - $P(D)$ is the probability of the data observed independent of the hypothesis. We need P(D) in order to make sure that our posterior probability is correctly placed somewhere between 0 and 1. > $P(D)$ is … totally unnecessary if all we care about is comparing the relative strength of two different hypotheses. … For these reasons, we often use the *proportional form* of `r glossary("Bayes’ theorem")`. $$P(H \mid D) \propto P(H) \times P(D \mid H)$$ {#eq-bayes-prop} > the posterior probability of our hypothesis is proportional to the prior multiplied by the likelihood. We can use this to compare two hypotheses by examining the ratio of the prior belief multiplied by the likelihood for each hypothesis using the *ratio of posteriors* formula: $$\frac{P(H_{1}) \times P(D \mid H_{1})}{P(H_{2}) \times P(D \mid H_{2})}$$ {#eq-posterior-ratio} > if the ratio is 2, then $H_{1}$ explains the observed data twice as well as $H_{2}$, and if the ratio is $\frac{1}{2}$, then $H_{2}$ explains the data twice as well as $H_{1}$. ## Building a Hypothesis Test Using the Ratio of Posteriors > The ratio of posteriors formula gives us the *posterior odds*, which allows us to test hypotheses or beliefs we have about data. > To better understand the posterior odds, we’ll break down the ratio of posteriors formula into two parts: the likelihood ratio, or the Bayes factor, and the ratio of prior probabilities. ### The Bayes Factor $$\frac{P(D \mid H_{1})}{P(D \mid H_{2})}$$ {#eq-bayes-factor} > What this ratio tells us is the likelihood of what we’ve seen given what *we* believe to be true compared to what *someone else* believes to be true. > The key here is that in Bayesian reasoning, we don’t worry about supporting our beliefs—we are focused on how well our beliefs support the data we observe. In the end, data can either confirm our ideas or lead us to change our minds. ### Prior Odds > So far we have assumed that the prior probability of each hypothesis is the same. This is clearly not always the case: a hypothesis may explain the data well even if it is very unlikely. $$\frac{P(H_{1})}{P(H_{2})}$$ > This ratio compares the probability of two hypotheses before we look at the data. When used in relation to the Bayes factor, this ratio is called the prior odds in our $H_{1}$ and written as $O(H_{1})$. This representation is helpful because it lets us easily note how strongly (or weakly) we believe in the hypothesis we’re testing. When this number is greater than 1, it means the prior odds favor our hypothesis, and when it is a fraction less than 1, it means they’re against our hypothesis. For example, $O(H_{1}) = 100$$ means that, without any other information, we believe $H_{1}$ is 100 times more likely than the alternative hypothesis. ### Posterior Odds $$\text{posterior odds} = O(H_{1})\frac{P(D \mid H_{1})}{P(D \mid H_{2})}$$ {#eq-posterior-odds} | Posterior odds | Strength of evidence | |----------------|-------------------------------------| | 1 to 3 | Interesting, but nothing conclusive | | 3 to 20 | Looks like we’re on to something | | 20 to 150 | Strong evidence in favor of $H_{1}$ | | > 150 | Overwhelming evidence | : Guidelines for Evaluating Posterior Odds {#tbl-guidelines-odds} ### Empty: Guidelines for Evaluating Posterior Odds ### Empty: Self-Diagnosing Rare Diseases Online ## Wrapping Up > In this chapter, you learned how to use the Bayes factor and posterior odds to compare two hypotheses. ## Exercises > Try answering the following questions to see how well you understand the Bayes factor and posterior odds. The solutions can be found at https://nostarch.com/learnbayes/. ### Exercise 16-1 > Returning to the dice problem, assume that your friend made a mistake and suddenly realized that there were, in fact, two loaded dice and only one fair die. How does this change the prior, and therefore the posterior odds, for our problem? Are you more willing to believe that the die being rolled is the loaded die? Let's recapitulate: > Suppose your friend has a bag with three six-sided dice in it, and one die is weighted so that it lands on 6 half the time. The other two are traditional dice whose probability of rolling a 6 is ⅙. Your friend pulls out a die and rolls 10 times, with the following results: $6, 1, 3, 6, 4, 5, 6, 1, 2, 6$ ::: {#def-bayes-factor} ##### Bayes Factor $$\frac{P(D \mid H_{1})}{P(D \mid H_{2})}$$ ::: It turned out that $H_{1}$ --- the loaded dice --- explain the data 3.77 times better than the fair dice with $H_{2}$. > However, this is true only if $H_{1}$ and $H_{2}$ are both just as likely to be true in the first place. But we know now for this exercise that there are two loaded dice in the bag and only one fair die, which means that each hypothesis was not equally likely. Based on the distribution of the dice in the bag, we know that these are the prior probabilities for each hypothesis: $$P(H_{1} = \frac{2}{3}); P(H_{2} = \frac{1}{3})$$ > From these, we can calculate the prior odds for $H_{1}$: $$\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{\frac{2}{3}}{\frac{1}{3}} = \frac{6}{3} = 2$$ > With our prior odds for H1, we can now compute our full posterior odds: $$\text{posterior odds} = O(H_{1}) = \frac{P(D \mid H_{1})}{P(D \mid H_{2})} = 2 \times 3.77 = 7.54$$ *** ::: {#lem-solution-16-1} Yes, I am now more willing to believe that the die being rolled is the loaded die! ::: *** ### Exercise 16-2 > Returning to the rare diseases example, suppose you go to the doctor, and after having your ears cleaned you notice that your symptoms persist. Even worse, you have a new symptom: vertigo. The doctor proposes another possible explanation, labyrinthitis, which is a viral infection of the inner ear in which 98 percent of cases involve vertigo. However, hearing loss and tinnitus are less common in this disease; hearing loss occurs only 30 percent of the time, and tinnitus occurs only 28 percent of the time. Vertigo is also a possible symptom of vestibular schwannoma, but occurs in only 49 percent of cases. In the general population, 35 people per million contract labyrinthitis annually. What is the posterior odds when you compare the hypothesis that you have labyrinthitis against the hypothesis that you have vestibular schwannoma? **Labyrinthitis**: $$P(D \mid H_{1}) = 0.98 (\text{vertigo}) \times 0.30 (\text{hearing loss}) \times 0.28 (\text{tinnitus}) = 0.082$$ **Vestibular schwannoma** $$P(D \mid H_{2}) = 0.49 (\text{vertigo}) \times 0.94 (\text{hearing loss}) \times 0.89 (\text{tinnitus}) = 0.41$$ **Bayes factor**: $$\frac{P(D \mid H_{1})}{P(D \mid H_{2})} = \frac{0.08}{0.48} = 0.17$$ **Including prior odds**: $$\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{\frac{35}{1,000,000}}{\frac{11}{1,000,000}} = \frac{35}{11} = 3.18$$ Based on prior information alone, a given person is about only about 3 times more likely to have labyrinthitis than vestibular schwannoma. Now let's compute the full posterior odds to see if the situation gets better: $$\text{prior odds} = O(H_{1}) = \frac{P(H_{1})}{P(H_{2})} = \frac{35}{11} \times 6 = 19,09$$ *** ::: {#lem-solution-16-2} The probability of having labyrinthitis is 19 times higher than vestibular schwannoma. ::: *** ::: {.callout-warning} Will Kurt has another result because he is calculating the labyrinthitis against the earwax impaction! :::