15 Making decisions: An introduction

So far, you have learnt to ask a RQ, identify different ways of obtaining data, design the study, collect the data describe the data, and summarise data graphically and numerically.

In this chapter, you will learn how decisions are made in science, so we can answer RQs. You will learn to:

  • explain the two broad reasons why differences are seen between sample statistics and population parameters.
  • explain how decisions are made in research.

15.1 Introduction

In Sect. 14.7, the NHANES data348 were numerically summarised. The sample mean direct HDL cholesterol concentration was different for smokers (\(\bar{x} = 1.31\)mmol/L) and for non-smokers (\(\bar{x} = 1.39\)mmol/L).

What does this difference between the sample means imply about the population means?

Two reasons could explain why the sample means are different:

  1. The population means are the same. The sample means are different because every sample is likely to be different (each possible sample includes different people), so, sometimes the sample means are different by chance. This is called sampling variation.

  2. Alternatively, the population means are different, and the sample means simply reflect this.

Similarly, in Sect. 14.7 the odds of being diabetic were different for smokers (0.181) and non-smokers (0.084). What does this difference between the sample odds imply about the population odds?

Again, two possible reasons could explain why the sample odds are different:

  1. The population odds are the same. The sample odds are different because every sample is likely to be different (each possible sample includes different people), so sometimes, the sample odds are different by chance. This is called 'sampling variation'.

  2. Alternatively, the odds are different in the population, and the sample odds simply reflect this.

In both situations (means; odds), the two possible explanations ('statistical hypotheses') have special names:

  1. There is no difference between the population parameters: this is the null hypothesis, or \(H_0\).
  2. There is a difference between the population parameters; this is the alternative hypothesis, or \(H_1\).

(The word hypothesis just means 'a possible explanation'.) A decision needs to be made about which of these two explanation is the most likely. However, because a sample is studied, conclusions about the population are never certain.

15.2 The need for making decisions

In research, decisions need to be made about population parameters based on sample statistics. The difficulty is that every sample is likely to be different (comprise different individuals from the population), and each sample will produce different summary statistics. This is called sampling variation.

Sampling variation refers to how much a sample estimate (a statistic) is likely to vary from sample to sample, because each sample is different.

However, sensible decisions can be (and are) made about population parameters based on sample statistics. For example, to determine if a pot soup is ready to serve, we don't have to consume the whole pot of soup (the 'population'); a sensible decision can be made from a small taste (the 'sample'). Likewise, in research sensible decisions about the population parameter can be made from the sample statistic.

To do this though, the process of how decisions are made needs to be articulated. In this chapter, the logic of making decisions is discussed.

To begin, consider the following scenario. Suppose I produce a standard pack of cards, and shuffle them well. The pack of cards can be considered a population.

A standard pack of cards has 52 cards, with four suits: spades and clubs (which are both black), and hearts and diamonds (which are both red).

Each suit has 13 denominations: 2, 3, 4, 5, 6, 7, 8, 9, 10, Jack (J), Queen (Q), King (K), Ace (A). Most packs also contain two jokers, but these special cards are not usually considered part of a standard pack.

Suppose I draw a sample of 15 cards from the pack, and notice that all are red cards. How likely is it that this would happen simply by chance? See the animation below. Is that evidence that the pack of cards is somehow unfair, or rigged?

Getting 15 reds cards out of 15 seems very unlikely, so perhaps you may conclude that the pack is unfair in some way. But importantly, how did you reach that decision? Your unconscious decision-making process may have worked like this:

  1. You assumed, quite reasonably, that this is a standard, well-shuffled pack of cards, so that half the cards are red and half the cards are black.
  2. Based on that assumption then, you, quite reasonably, expected about half the cards in the sample of 15 to be red, and about half to be black. You wouldn't necessarily expect to see exactly half red and half black, but you'd probably expect something close to that.
  3. But what you observed was nothing like that: All 15 cards were red. Since what you observed ('all red cards') was not like what you were expecting ('about half red cards'), the 15 cards in my hand contradict what you were expecting, based on your assumption of a fair pack... so your assumption of a fair pack is probably wrong.

Of course, getting 15 red cards in a row is possible... but very unlikely349. For this reason, we would probably conclude that the most likely explanation is that the pack is not a fair pack.

You probably didn't consciously go through this process, but it does seem reasonable. This process of decision making is similar to the process used in research.

15.3 How decisions are made

Based on the ideas in the last section, a formal process of decision making in research can be described as follows.

TABLE 15.1: The decision-making process
Assumption Make a reasonable assumption about the value of a population parameter
Expectation Based on this assumption, describe what values of the sample statistic might reasonably be observed
Observation Observe the sample statistic. Then, if the observed sample statistic is: ...unlikely to happen by chance, it contradicts the assumption.
Observation Observe the sample statistic. Then, if the observed sample statistic is: ...likely to happen by chance, it is consistent with the assumption.

To expand:

  1. Assumption: Make a reasonable assumption about the population, such as the value of a population parameter, or state a value for the population parameter to be confirmed.

  2. Expectation: Based on this assumption, describe what might be observed in the sample, such as values of the sample statistic that might reasonably be observed from all possible samples.

  3. Observation: If the observed sample statistic is:

    • unlikely to happen by chance, it contradicts the assumption about the population parameter, and the assumption is probably wrong. The evidence suggests that the assumption is wrong (but it is not certainly wrong).
    • likely to happen by chance, it is consistent with the assumption about the population parameter, and the assumption may be correct. No evidence exists to suggest the assumption is wrong (though it may be wrong).

This is one way to describe the formal process of decision making in science (Fig. 15.1).

A way to make decisions

FIGURE 15.1: A way to make decisions

This approach is similar to what we use every day without really thinking about it For example, suppose I ask my son to brush his teeth,350 and later I want to decide if he really did brush his teeth.

  1. Assumption: I assume my son brushed his teeth (because I told him to).
  2. Expectation: Based on that assumption, I expect to find a damp toothbrush when I check later.
  3. Observation: When I check later, I observe a dry toothbrush. The evidence seems to contradict my assumption, as I did not find what I expected, so my assumption is probably false: He probably didn't brush his teeth.

Of course, I may have made the wrong decision: He may have brushed his teeth, but his brush is now dry (he may have dried his brush with a hair dryer; he's that sort of kid). However, based on the evidence, quite probably he has not brushed his teeth.

The situation may have ended differently:

  1. Assumption: I assume my son brushed his teeth (because I told him to).
  2. Expectation: Based on that assumption, I expect to find a damp toothbrush when I check later.
  3. Observation: When I check later, I observe a damp toothbrush. The evidence seems consistent with my assumption, as I found what I expected, so my assumption is probably true: He probably did brush his teeth.

Again, I may be wrong: He may have just ran his toothbrush under a tap (again, it wouldn't surprise me). I don't have any evidence that he didn't brush his teeth, though; I can hardly get him into trouble.

This logic underlies most decision making in science351.

Example 15.1 (The decision-making process) Consider the cards example from Sect. 15.2 again. The formal process might look like this:

  1. Assumption: Initially assume the pack is fair and well-shuffled pack of cards (you have no evidence to doubt this).

    In other words, the proportion of red cards is 0.5 (the value of the parameter).

  2. Expectation: Based on this assumption, roughly equal (but not necessarily exactly) equal numbers of red and black cards would be expected in a sample of 15 cards.

    In other words, the proportion of red cards in any sample is expected to be close to, but maybe not exactly, 0.5 (the value of the statistic).

  3. Observation: Suppose I then deal 15 cards, and all 15 are red cards.

This seems unlikely to occur if the pack is fair and well-shuffled; the data seem inconsistent with what I was expecting based on the assumption (Fig. 15.2). The evidence suggests that the assumption is probably false.

Of course, getting 15 red cards out of 15 is not impossible, so I may be wrong... but it is very unlikely. Based on the evidence, concluding that a problem exists with the pack of cards seems reasonable.

A way to make decisions for the cards example

FIGURE 15.2: A way to make decisions for the cards example

15.4 Making decisions in research

Let's think about each step in the decision-making process (Fig. 15.1) individually.

15.4.1 Assumption about the population parameter

Usually a reasonable assumption can be made about the population parameter. For example:

  • We might assume that no difference exists between the parameter for two groups in the population, since we don't have any evidence yet to say there is a difference.
    For example, we might assume that the mean HDL cholesterol (or the odds of a diabetes diagnosis) is the same for current smokers and non-smokers in the population, for the NHANES data. If we already knew there was a difference, why would we be performing a study to see if there is a difference?

  • We might be interested in testing a claim, or evaluating a benchmark, about a population parameter, to determine if the evidence supports this claim or benchmark,

These assumptions about the population parameter are called null hypotheses.

Example 15.2 (Assumptions about the population) Most dental associations, such as the American Dental Association and the Australian Dental Association, recommend brushing teeth for two minutes. One study352 recorded the tooth-brushing time for 85 uninstructed schoolchildren (11 to 13 years old) from England.

We could assume the population mean tooth-brushing time in the population ('school children (11 to 13 years old) from England') is two minutes, as recommended. After all, we don't have evidence to suggest any other value for the mean. A sample can then be obtained to determine if the sample mean is consistent with, or contradicts, this assumption.

15.4.2 Expectations of sample statistics

Having made an assumption about the population parameter, the second step is to determine what values to expect from the sample statistic, based on this assumption.

Since every sample is likely to be different ('sampling variation', the value of the sample statistic depends on which of the possibe samples we end up with: the sample statistic is likely to be different for every sample.

Think about the cards in Sect. 15.2. Assuming a fair pack, then half the cards are red in the population (the pack of cards), so the population proportion is assumed to be \(p=0.5\).

In a sample of 15 cards, what values could be reasonably expected for the sample proportion \(\hat{p}\) of red cards (the statistic)? If samples of size 15 were repeatedly taken, the sample proportion of red cards would vary from hand to hand, of course.

How much would \(\hat{p}\) vary from sample to sample? Perhaps 15 red cards out of 15 cards happens reasonably frequently. Or perhaps it doesn't. How could we find out? We could:

  • Use mathematical theory to determine how likely it is that we would get 15 red cards out of 15 cards.
  • We could repeatedly shuffle a pack of cards, and repeatedly deal 15 cards many hundreds or thousands of times, then compute how often we get 15 red cards of out 15 cards.
  • More reasonably, we could simulate (using a computer) dealing 15 cards many hundreds or thousands of times, and count how often we get 15 red cards of out 15 cards.

The third option is the most practical... To begin, suppose we simulated only ten hands of 15 cards each; the animation below shows the sample proportion of red cards from ten repetitions. Not one of those ten hands produced 15 red cards in 15 cards.

Suppose we repeated this for hundreds of hands of 15 cards, and for each hand we recorded the sample proportion of cards that were red. The proportion of red cards would vary from sample to sample ('sampling variation'), and we could record the proportion of red cards from each of those hundreds of hands.

For these hundreds of sample proportions, we could draw a histogram; for example, the animation below shows a histogram of the sample proportions from 1000 repetitions of a hand of 15 cards.

This histogram shows how we might expect the sample proportions \(\hat{p}\) to vary from sample to sample, when the population proportion of red cards is \(p=0.5\).

We can see that observing 15 red cards out of 15 cards is quite rare: it never happened once in the 1000 simulations.

15.4.3 Observations about our sample

From this histogram, based on a simulation of one thousand hands, we could conclude that we would almost never find 15 red cards in 15 cards... if the assumption of a fair pack was true. But we did find 15 red cards in 15 cards... so the assumption ('a fair pack') is probably wrong.

What if we had observed 4 red cards in a hand of 15 cards (a sample proportion of \(\hat{p} = 4/15 = 0.267\)), rather than 15 red cards out of 15? The conclusion is not quite so obvious then: these values of \(\hat{p}\) are uncommon, but they certainly do happen when \(p=0.5\). In these situations, a more sophisticated approach for making a decision is needed.

Special tools are needed to describe what to expect from the sample statistic after making assumptions about the population parameter. These special tools are discussed in the next chapter.

Example 15.3 (Sampling variation) Most dental associations, such as the American Dental Association and the Australian Dental Association, recommend brushing teeth for two minutes. One study353 recorded the tooth-brushing time for 85 uninstructed schoolchildren from England (11 to 13 years old).

The sample mean tooth-brushing time may or may not be two minutes. Of course, every possible sample of 85 children will include different children, and so produce a different sample mean \(\bar{x}\). Even if the population mean toothbrushing time really is two minutes (\(\mu=2\)), the sample mean probably won't be exactly two minutes, because of sampling variation.

We could assume the population mean tooth-brushing time is two minutes (\(\mu=2\)). If this assumption is true, we then could describe what values of the sample statistic \(\bar{x}\) to expect. Then, after obtaining a sample and computing the sample mean, we could determine if the sample mean seems consistent with the assumption of two minutes, or whether it seems to contradict this assumption.

15.5 Tools for describing sampling variation

As we have observed previously, making decisions about population parameters based on a sample statistic can be difficult: Every sample is likely to be different, and can produce a different value of the sample statistic.

In this chapter, though, a process for making decisions has been studied (Fig. 15.1). To apply this process to research, we need to describe how sample statistics vary from sample to sample (sampling variation). To do so, some of those tools are discussed in the following chapters:

  • Tools to describe the population and the sample: Chap. 17.
  • Tools to describe how sample statistics vary from sample to sample (sampling variation), and hence what to expect from the sample statistic: Chap. 18.
  • Tools to describe the random nature of what happens with sample statistics, and so determine if the sample statistic is consistent with the assumption: Chap. 16.

15.6 Summary

Decisions are often made by first making an assumption about the population parameter, which leads to an expectation of what might occur in the sample statistics. We can then make observations about our sample, to see if it seems to support or contradict the initial assumption.

15.7 Quick review questions

  1. True or false: Parameters describe populations.
  2. True or false: Both \(\bar{x}\) and \(\mu\) are statistics.
  3. True or false: The value of a statistic is likely to be different in every sample.
  4. True or false: Sampling variation refers to how the value of a statistic varies from sample to sample.
  5. True or false: The initial assumption is made about the sample statistic.
  6. True or false: The variation in statistics from sample to sample is called sampling variation
  7. True or false: If the sample results seem inconsistent with what was expected, then the assumption about the population is probably true.
  8. True or false: In the sample, we know exactly what to expect.
  9. True or false:Hypotheses are made about the population.

Progress:

15.8 Exercises

Selected answers are available in Sect. D.15.

Exercise 15.1 Suppose you are playing a die-based game, and your opponent rolls a 6 ten times in a row.

  1. Do you think there is a problem with the die?
  2. Explain how you came to this decision.

Exercise 15.2 In a 2012 advertisement, an Australian pizza company claimed that their 12-inch pizzas were 'real 12-inch pizzas', unlike another brand.354

  1. What is a reasonable assumption to make to test this claim?
  2. The claim is based on a sample of 125 pizzas, for which the sample mean pizza diameter was \(\bar{x} = 11.48\) inches. What are the two reasons why the sample mean is not 12-inches?
  3. Does the claim appear to be supported by, or contradicted by, the data? Why?
  4. Would your conclusion change if the sample mean was \(\bar{x} = 11.25\) inches, rather than 11.48 inches? Does the claim appear to be supported by, or contradicted by, the data? Why?
  5. Does your answer depend on the sample size? For example, is observing a sample mean of 11.25 inches from a sample of size 10 equivalent to observing a sample mean of 11.25 inches from a sample of size 125?