18 When Data Doesn’t Convince

In this chapter, we’ll take a deeper look at problems where the data doesn’t convince people in the way we expect it to. In the real world, these situations are fairly common.

18.1 A Psychic Friend Rolling Dice

18.1.1 Comparing Likelihoods

we’ll start by looking at the Bayes factor, assuming for now that the ´r glossary(“prior odds”)` for each hypothesis are equal.

18.1.2 Incorporating Prior Odds

In most cases in this book where the likelihood alone gives us strange results, we can solve the problem by including our prior probabilities.

In most of our previous problems, we’ve corrected nonintuitive posterior results by adding a sane prior. We’ve added a pretty extreme prior against your friend being psychic, but our posterior odds are still strongly in favor of the hypothesis that they’re psychic.

18.1.3 Considering Alternative Hypotheses

The issue here is that we don’t want to believe your friend is psychic. If you found yourself in this situation in real life, it’s likely you would quickly come to some alternative conclusion. You might come to believe that your friend is using a loaded die that rolls a certain value about 90 percent of the time, for example. This represents a third hypothesis.

It’s important to understand that a hypothesis test compares only two explanations for an event, but very often there are countless possible explanations. If the winning hypothesis doesn’t convince you, you could always consider a third one.

18.2 Arguing with Relatives and Conspiracy Theorists

How can we change someone else’s (or our own) beliefs even when more data doesn’t change anything?

The danger of nonfalsifiable beliefs in Bayesian reasoning isn’t just that they can’t be proved wrong—it’s that they are strengthened even by evidence that seems to contradict them. Rather than persisting in trying to convince you, your friend should have first asked, “What can I show you that would change your mind?” If your reply had been that nothing could change your mind, then your friend would be better off not presenting you with more evidence.

18.3 Wrapping Up

Although the Bayes factor is a competition between two ideas, it’s quite possible that there are other, equally valid, hypotheses worth testing out.

Other times, we find that two hypotheses explain the data equally well; … When this is the case, only the prior odds ratio for each hypothesis matters. This also means that acquiring more data in those situations will never change our beliefs, because it will never give either hypothesis an edge over the other. In these cases, it’s best to consider how you can alter the prior beliefs that are affecting the results.

In more extreme cases, we might have a hypothesis that simply refuses to be changed. This is like having a conspiracy theory about the data. When this is the case, not only will more data never convince us to change our beliefs, but it will actually have the opposite effect. If a hypothesis is not falsifiable, more data will only serve to make us more certain of the conspiracy.

18.4 Exercises

Try answering the following questions to see how well you understand how to deal with extreme cases in Bayesian reasoning. The solutions can be found at https://nostarch.com/learnbayes/.

18.4.1 Exercise 18-1

When two hypotheses explain the data equally well, one way to change our minds is to see if we can attack the prior probability. What are some factors that might increase your prior belief in your friend’s psychic powers?

Solution 18.1 I would change the rules for my friend to demonstrate his/her psychic power to me. For instance: - Instead of tossing his coin I want it to do with a coin of my own. - Instead of tossing the coin on a table I will catch the coin in the air. (I read that then it is much more difficult to control the coin.) - Or I will ask him other things like what figure I am thinking about, etc.

18.4.2 Exercise 18-2

An experiment claims that when people hear the word Florida, they think of the elderly and this has an impact on their walking speed. To test this, we have two groups of 15 students walk across a room; one group hears the word Florida and one does not. Assume $H_{1}$ = the groups don’t move at different speeds, and $H_{2}$ = the Florida group is slower because of hearing the word Florida. Also assume the Bayes factor from Equation 16.4:

\[BF = \frac{P(D \mid H_{1})}{P(D \mid H_{2})}\]

The experiment shows that $H_{2}$ has a Bayes factor of 19. Suppose someone is unconvinced by this experiment because $H_{2}$ had a lower prior odds. What prior odds would explain someone being unconvinced and what would the BF need to be to bring the posterior odds to 50 for this unconvinced person?

Warning

This time my answer of 25 to 50 was completely wrong. I calculated wrongly the unconvincing prior odds between 1 to 3 taking erroneously values of Table 16.1 and did not take into account to equalize the fraction $\frac{1}{19}$

Solution 18.2 \[50 = \frac{1}{19} \times x = 50 \times 19 = 950\]

Now suppose the prior odds do not change the skeptic’s mind. Think of an alternate $H_{3}$ that explains the observation that the Florida group is slower. Remember if $H_{2}$ and $H_{3}$ both explain the data equally well, only prior odds in favor of $H_{3}$ would lead someone to claim $H_{3}$ is true over $H_{2}$, so we need to rethink the experiment so that these odds are decreased. Come up with an experiment that could change the prior odds in $H_{3}$ over $H_{2}$

Solution 18.3 The majority of the Florida group members might be older (adult students) and therefore walking slower. Or they have some handicapped people in it, so that the average of the walking speed of the whole group is reduced.

The books solution was shorter people walking more slowly.

--- engine: knitr --- # When Data Doesn't Convince > In this chapter, we’ll take a deeper look at problems where the data doesn’t convince people in the way we expect it to. In the real world, these situations are fairly common. ## A Psychic Friend Rolling Dice ### Comparing Likelihoods > we’ll start by looking at the `r glossary("Bayes factor")`, assuming for now that the ´r glossary("prior odds")` for each hypothesis are equal. ### Incorporating Prior Odds > In most cases in this book where the likelihood alone gives us strange results, we can solve the problem by including our `r glossary("Prior probability", "prior probabilities")`. > In most of our previous problems, we’ve corrected nonintuitive posterior results by adding a sane prior. We’ve added a pretty extreme prior against your friend being psychic, but our posterior odds are still strongly in favor of the hypothesis that they’re psychic. ### Considering Alternative Hypotheses > The issue here is that *we don’t want to believe your friend is psychic*. If you found yourself in this situation in real life, it’s likely you would quickly come to some alternative conclusion. You might come to believe that your friend is using a loaded die that rolls a certain value about 90 percent of the time, for example. This represents a *third* hypothesis. > It’s important to understand that a hypothesis test compares only two explanations for an event, but very often there are countless possible explanations. If the winning hypothesis doesn’t convince you, you could always consider a third one. ## Arguing with Relatives and Conspiracy Theorists > How can we change someone else’s (or our own) beliefs even when more data doesn’t change anything? > The danger of nonfalsifiable beliefs in Bayesian reasoning isn’t just that they can’t be proved wrong—it’s that they are strengthened even by evidence that seems to contradict them. Rather than persisting in trying to convince you, your friend should have first asked, “What can I show you that would change your mind?” If your reply had been that *nothing* could change your mind, then your friend would be better off not presenting you with more evidence. ## Wrapping Up > Although the Bayes factor is a competition between two ideas, it’s quite possible that there are other, equally valid, hypotheses worth testing out. > Other times, we find that two hypotheses explain the data equally well; … When this is the case, only the prior odds ratio for each hypothesis matters. This also means that acquiring more data in those situations will never change our beliefs, because it will never give either hypothesis an edge over the other. In these cases, it’s best to consider how you can alter the prior beliefs that are affecting the results. > In more extreme cases, we might have a hypothesis that simply refuses to be changed. This is like having a conspiracy theory about the data. When this is the case, not only will more data never convince us to change our beliefs, but it will actually have the opposite effect. If a hypothesis is not falsifiable, more data will only serve to make us more certain of the conspiracy. ## Exercises > Try answering the following questions to see how well you understand how to deal with extreme cases in Bayesian reasoning. The solutions can be found at https://nostarch.com/learnbayes/. ### Exercise 18-1 > When two hypotheses explain the data equally well, one way to change our minds is to see if we can attack the prior probability. What are some factors that might increase your prior belief in your friend’s psychic powers? *** ::: {#lem-solution-18-1} I would change the rules for my friend to demonstrate his/her psychic power to me. For instance: - Instead of tossing his coin I want it to do with a coin of my own. - Instead of tossing the coin on a table I will catch the coin in the air. (I read that then it is much more difficult to control the coin.) - Or I will ask him other things like what figure I am thinking about, etc. ::: *** ### Exercise 18-2 > An experiment claims that when people hear the word Florida, they think of the elderly and this has an impact on their walking speed. To test this, we have two groups of 15 students walk across a room; one group hears the word Florida and one does not. Assume $H_{1}$ = the groups don’t move at different speeds, and $H_{2}$ = the Florida group is slower because of hearing the word Florida. Also assume the Bayes factor from @eq-bayes-factor: $$BF = \frac{P(D \mid H_{1})}{P(D \mid H_{2})}$$ > The experiment shows that $H_{2}$ has a Bayes factor of 19. Suppose someone is unconvinced by this experiment because $H_{2}$ had a lower prior odds. What prior odds would explain someone being unconvinced and what would the BF need to be to bring the posterior odds to 50 for this unconvinced person? ::::: {.callout-warning} This time my answer of 25 to 50 was completely wrong. I calculated wrongly the unconvincing prior odds between 1 to 3 taking erroneously values of @tbl-guidelines-odds and did not take into account to equalize the fraction $\frac{1}{19}$ *** ::: {#lem-solution-18-2a} $$50 = \frac{1}{19} \times x = 50 \times 19 = 950$$ ::: *** ::::: > Now suppose the prior odds do not change the skeptic’s mind. Think of an alternate $H_{3}$ that explains the observation that the Florida group is slower. Remember if $H_{2}$ and $H_{3}$ both explain the data equally well, only prior odds in favor of $H_{3}$ would lead someone to claim $H_{3}$ is true over $H_{2}$, so we need to rethink the experiment so that these odds are decreased. Come up with an experiment that could change the prior odds in $H_{3}$ over $H_{2}$ *** ::: {#lem-solution-18-2b} The majority of the Florida group members might be older (adult students) and therefore walking slower. Or they have some handicapped people in it, so that the average of the walking speed of the whole group is reduced. The books solution was shorter people walking more slowly. ::: ***