2.1 The concept of probability

Let us begin with the following thought experiment:

Assume that you are watching a contestant compete in the first round of the TV show “Who wants to be a millionaire?” he is asked to answer a very simple question: What is the last name of the brothers who are credited with inventing the world’s first successful motor-operated airplane?

  • What is the probability that the contestant answers this question correctly and, hence, survives the first round?

Unless you have:

  1. watched this particular contestant participate in this show many times

  2. seen him asked this same question each time

  3. and computed the relative frequency with which he gives the correct answer,

you need to answer this question as a Bayesian!

Uncertainty about the event “survival” needs to be expressed as a “degree of belief” informed both by information (or “data”) on the skill of the particular participant, and how much he knows about inventors, and (possibly) prior knowledge on his performance in other game shows. Of course, your prior knowledge of the contestant may be minimal, or it may be very informed. Either way, your final answer remains a degree of belief held about an uncertain (and inherently unrepeatable) state of nature.

The point of this hypothetical, light-hearted scenario is simply to highlight that a key distinction between the “Frequentist” and Bayesian approaches to inference is not the use (or nature) of prior information, but simply the manner in which probability is used. To the Bayesian, probability is the mathematical construct used to quantify uncertainty about an unknown state of nature, conditional on observed data and prior knowledge about the context in which that state of nature occurs. To the Frequentist, probability is linked intrinsically to the concept of a repeated experiment, and the relative frequency with which a particular outcome occurs, conditional on that unknown state. This distinction remains key whether the Bayesian chooses to be informative or subjective in the specification of prior information, or chooses to be noninformative or objective.

Frequentists consider probability as a physical phenomenon, like mass or wavelength, whereas Bayesians stipulate that probability lives in the mind of scientists (Parmigiani and Inoue 2008).

It seems that the understanding of the concept of probability for the common human being is more associated with “degrees of belief” rather than relative frequency. Peter Diggle, President of The Royal Statistical Association (2014-2016), said in an interview:

“A different trend which has surged upwards in statistics during Peter’s career is the popularity of Bayesian statistics. Does Peter consider himself a Bayesian? Well, he replies, you can’t not believe in Bayes’ theorem because it’s true. But that doesn’t make you a Bayesian in the philosophical sense. When people are making personal decisions – even if they don’t formally process Bayes’ theorem in their mind – they are adapting what they think they should believe in response to new evidence as it comes in. Bayes’ theorem is just the formal mathematical machinery for doing that.”

However, I should say that psychological experiments suggest that human beings suffer from anchoring, that is, a cognitive bias that causes us to rely too heavily on the previous information (prior) such that the updating process (posterior) due to new information (likelihood) is low compared to the Bayes’ rule (Kahneman 2011).

References

Kahneman, Daniel. 2011. Thinking, Fast and Slow. Macmillan.
Parmigiani, G., and L. Inoue. 2008. Decision Theory Principles and Approaches. John Wiley & Sons.