6 Overview of internal validity

So far, you have learnt to ask a RQ, select a study type, and find a sample to study. In this chapter, you will learn how to ensure that the conclusions are logical and sound. You will learn to:

  • identify reasons why the value of the response variable varies.
  • identify extraneous, confounding and lurking variables.
  • understand how chance impacts the values of the response variable.

6.1 Introducing internal validity

Well-designed studies allow the researchers to focus on the relationship of interest: between the explanatory and response variables. Studies for which this is true have good internally validity. Such studies eliminate, or account for, other explanations for the variation in the values of the response variable.

Definition 6.1 (Internal validity) Internally validity refers to the extent to which a cause-and-effect relationship can be established in a study.

A study with high internal validity shows that the changes in the response variable can be attributed to changes in the explanatory variables; other explanations have been ruled out.

Studies with high internal validity show that changes in the response variable can confidently be related to changes in the explanatory variable in the group that was studied; the possibility of other explanations has been minimised.

In contrast, studies with low internal validity leave open other possibilities, apart from changes in value of the explanatory variable, to explain changes in the value of the response variable. In general, well-designed experimental studies are more likely to be internally valid than observational studies.

Ideally, all studies should be designed to be internally valid as far as possible. For this reason, internal validity is studied in more detail in Chap. 7 (for experimental studies) and Chap. 8 (for observational studies).

6.2 Values of the response variable vary

Consider the letter-typing RQ again (from Example 5.5):

For students at a large course at a university, is the average typing speed (in words per minute) the same for those aged under \(25\) ('younger') and \(25\) or over ('older')?

The typing speeds (the response variable) of the many individuals will vary: every student in the study recording exactly the same typing speed is highly unlikely. The variation in the typing speeds (Fig. 6.1) may be due to:

  • The explanatory variable (Sect. 6.4): The values of the explanatory variable may influence the values of the response variable. Of course, they may not either; the purpose of the study is to find if, or to what extent, this is true. In this example, the explanatory variable is the age group of the student.
  • Other variables, called extraneous variables (Sect. 6.5): Other variables (apart from the explanatory variable) may influence the response variable (perhaps more so than the explanatory variable) such as 'sex', or 'whether or not the person wears glasses'. The impact of these variables can be accommodated if the study is designed appropriately.
  • Chance (or randomness or natural variation) (Sect. 6.6): Even the same person doing the same thing repeatedly under the same conditions will not record exactly the same typing speed every attempt. This is unavoidable, but good research design can minimize the size of this variation.

Research design to maximize internal validity is concerned with planning ways to identify extraneous variables, and eliminating (as far as possible) anything that obscures the relationship between the response and explanatory variables.

Example 6.1 (Design) The typing-speed study could be poorly designed. Suppose females were always asked to use their dominant hand, and males always asked to use their non-dominant hand. Females would probably have a faster average time, simply because they use their dominant hands. Using this research design would produce a study with poor internal validity.

Other factors can influence the values of the response variable, besides the relationship of interest

FIGURE 6.1: Other factors can influence the values of the response variable, besides the relationship of interest

The purpose of the study is to explore the relationship between the response variable and the explanatory variable... but other sources of variation can obscure that relationship. An internally valid study helps establish the relationship between the response and explanatory variables, and eliminates other explanations for the relationship as far as possible (Fig. 6.1).

Definition 6.2 (Research design) Research design refers to the decisions made by the researchers to maximise external validity and internal validity.

Internal validity is one of the most important properties of scientific studies, and is important for reasoning about evidence more generally. Designing studies to maximise internal validity is the focus of Chaps. 7 (experimental studies) and 8 (observational studies). This chapter provides an overview of internal validity.

Data collection is often tedious, time consuming and expensive. You usually get one chance to collect data, but data can be analysed as many times as necessary. Design the study properly the first time!

6.3 Bias

Bias refers to any misrepresentation that can lead to a false conclusion. Bias compromises the results or inferences in a study, which may lead to inaccurate conclusions. Bias can be introduced at any part of the research process, including the sampling (Sect. 5.11).

Definition 6.3 (Bias) Bias refers to anything systematic which may cause differences between sample statistics and population parameters.

Bias may occur as a consequence of research design (poor internal validity), data collection (selection bias; Sect. 5.7), analysis, or interpretation of results (Chap. 9). This book only discusses a small number of possible biases.

6.4 Impact of changes in the explanatory variable

Changes in the values of the explanatory variable may be associated with changes in the values of the response variable. However, it may not; after all, determining if a relationship exists between the response and explanatory variables (or the extent of this relationship) is the purpose of the study. If nothing else influenced the values of the response variable, life would be easy: any change of a given size in the value of the explanatory variable would always result in a change of the same size in the value of the response variable.

Example 6.2 (Explanatory variable) In the typing-speed study (Sect. 6.2), the explanatory variable is the age group of the person. If nothing else influenced typing speed, every student under \(25\) record the same typing speed every time, and every student \(25\) or older would record the same typing speed every time. This is clearly unreasonable.

6.5 Importance of identifying extraneous variables

Other variables (besides the explanatory variable) almost certainly exist which are associated with changes in the value of the response variable. These are called extraneous variables.

Definition 6.4 (Extraneous variable) An extraneous variable is any variable associated with the response variable, but is not the explanatory variable.

Example 6.3 In the typing-speed study (Chap. 6.2), potential extraneous variables may include age, the presence or absence of certain medical conditions, the level of familiarity with computers, whether the person wears glasses, etc.

All extraneous variables are, by definition, related to the response variable. They may or may not also be associated with the explanatory variable. Extraneous variables may have other names too (Fig. 6.4), though these names are used inconsistently in research.

Extraneous variable also related to the explanatory variable are called confounding variables (or a confounders); see Fig. 6.2 (left panel). A confounding variable can obscure the true relationship between the response and explanatory variables (i.e., confounding variables can bias the results).

Definition 6.5 (Confounding variable) A confounding variable (or a confounder) is an extraneous variable associated with the response and explanatory variables.

Definition 6.6 (Confounding) Confounding is when a third variable influences the observed relationship between the response and explanatory variable.

A relationship between the response and explanatory variables may be apparent, but only because both of these variables are associated with the confounding variable (Fig. 6.2).

Example 6.4 (Confounding variables) People who carry cigarette lighters are more likely to get lung cancer. The reason this relationship exists, however, is because of a confounding variable. 'Whether or not the person is a smoker' is probably associated with both the response and explanatory variables:

  • A smoker is more likely to carry a cigarette lighter (the explanatory variable) than a non-smoker; and
  • A smoker is more likely to develop lung cancer (the response variable) than a non-smoker.

Consider this RQ:

Among university students, is the percentage of females who know their own blood pressure the same as the percentage of males who know their own blood pressure?

For this RQ, the explanatory variable is the sex of person, and the response variable is whether a student knows their own blood pressure. A potential confounding variable is 'The program of study', since this is (potentially) related to both the response and explanatory variables:

  • 'Program of study' is related to sex (the explanatory variable): a higher percentage of females study nursing, while a greater percentage of males study engineering (at least, in Australia).
  • 'Program of study' is related to knowing your blood pressure (the response variable): nursing students probably practice taking each others blood pressures so probably know theirs, whereas engineering students do not.

Managing confounding is very important, as ignoring confounding can completely change the relationship between the response and explanatory variables (see Sect. 14.6) and hence can compromise internal validity. Managing confounding is discussed in Sects. 7.2 and  8.2.

Confounding variables (left) are extraneous variables associated with the response and explanatory variables. Lurking variables (right) are associated with the response and explanatory variables, but are not recorded

FIGURE 6.2: Confounding variables (left) are extraneous variables associated with the response and explanatory variables. Lurking variables (right) are associated with the response and explanatory variables, but are not recorded

If the values of potential confounding variables are recorded, their impact can be managed. However, sometimes the values of the confounding variables are not recorded (perhaps due to poor design); then, they are called lurking variables (Fig. 6.2, right panel). Lurking variables can lead to wrong conclusions (for example, see Sect. 14.6).

Definition 6.7 (Lurking variable) A lurking variable is an extraneous variable associated with the response and explanatory variables (that is, is a confounding variable), but whose values are not recorded in the study data.

Example 6.5 (Lurking variables) A study (Joiner (1981); Wilson Jr (1952)) examined the strength of plastic parts, to determine if the time in the production mould influenced the strength of the part (p. 55--56):

Hot plastic was introduced in the mold, pressed for \(10\) seconds, and removed. Another batch was then introduced into the same mold, pressed for \(20\) seconds, and so on, the time increasing with each batch.

Greater time in the mould (explanatory variable) was found to be associated with greater plastic strength (response variable). However, mould temperature was a lurking variable: it was associated with both the response and explanatory variables:

  • Higher mould temperatures (the lurking variable) were associated with greater strength (the response variable); and
  • Higher mould temperatures (the lurking variable) were experienced by later batches with longer mould times (the explanatory variable), since the mould was hotter for the later batches.

The cause of the greater strength was not the time in the mould; it was the higher temperature experienced by the later moulds (Fig. 6.3).

An example of a lurking variable. Left: The relationship as originally understood. Right: the relationship after the lurking variable was eventually exposed.

FIGURE 6.3: An example of a lurking variable. Left: The relationship as originally understood. Right: the relationship after the lurking variable was eventually exposed.

To clarify the language (Fig. 6.4):

  • Extraneous variables are, by definition, always associated with the response variable. If they are not recorded, and so the researchers are unaware of them, they become part of unexplained chance.
  • Extraneous variables are called confounding variables if they are also related to the explanatory variable.
  • Confounding variables are called lurking variables, if they are not recorded.

These terms are not always used consistently by all researchers, but the ideas are important nonetheless.

The relationships between extraneous, confounding and lurking variables

FIGURE 6.4: The relationships between extraneous, confounding and lurking variables

To avoid lurking variables, researchers generally collect lots of information about the individuals in the study (such as age and sex if the study involves people) and circumstances of the individuals in the study (such as the temperature at the time of data collection) that may be relevant, in case they are confounding variables.

Example 6.6 (Low internal validity) L. M. Larson et al. (2021) reviewed numerous studies that used double-fortified salt to manage iodine and iron deficiencies. They concluded that the internal validity of studies was 'generally weak' (p. 265) die, in part, to 'unaccounted confounders' (i.e., lurking variables).

6.6 Chance and variation in the response variable

Chance (or natural) variation refers to variation that cannot otherwise be explained: even repeating a study exactly the same way every time on the same individuals will not always produce the same values of the response variable.

Chance variation makes the influence of the explanatory variable (which we want to study) hard to detect, so minimising chance variation is important. Minimising the amount of the chance variation requires using good design principles, and measuring as many other extraneous variables that may explain variation in the response variable as is reasonable.

Chance can impact the values of the response variable in different ways:

  • Each individual can produce different values of the response variable each time the response variable is measured (within-individuals variation); and
  • Each individual in the study can produce different values of the response variable compared to other individuals (between-individuals variation):

We need different strategies to understand each of these sources of variation:

  • To estimate the amount of variation within individuals: many observations are needed from each unit of analysis (individual).
  • To estimate amount of variation between individuals: many units of analysis (individuals) are needed.

Since between-individual variation is usually more variable (i.e., larger variation) than the within-individual variation, using many individuals is usually more important than using a smaller number of individuals many times each.

Consider the typing-speed study (Sect. 6.2) again. What are the advantages and disadvantage of measuring the typing speed for:

  1. one female \(30\) times?
  2. \(30\) different females once each?
  3. \(10\) different females, three times each?
  1. We would learn a lot about that female... but very little about females in general.
  2. We would learn a lot about females in general... but have one measurement from each. Since we might expect that the same person might produce similar (not necessarily identical) typing speed, this is not really a problem.
  3. We would learn a lot about females in general... and a little about each female too.

6.7 Chapter summary

In a research study, the main interest is usually the relationship between a response variable and explanatory variable. Well-designed studies that allow the researchers to focus on this relationship have good internally validity. Such studies eliminate, or account for, other explanations for the variation in the values of the response variable.

However the values of the response variable can be influenced by things other than the explanatory variable: extraneous variable (other variables that aren't of primary interest), and chance.

Some extraneous variables are also related to the explanatory variable, and are called confounding variables (and are lurking variables if not recorded). If the research design makes it difficult to separate the relationship between the response and explanatory variable from other possible causes, the study has poor internal validity.

6.8 Quick review questions

Martnes and Bere (2023) (p. 1) compared the

...time spent riding and exercise intensity when... riding an electric-assisted bicycle with cargo (\(30\) kg) and without cargo...

In their study, they recorded the age, height, weight, and resting metabolic rate of all subjects who completed the \(4.5\) km ride. Subjects were allocated by the researchers to ride both with and without cargo.

Are the following statements true or false?

  1. The explanatory variable is the age of the subjects.
  2. The height of the subjects is a lurking variable.
  3. The explanatory variable is whether the electric-assisted bike is ridden with or without cargo.
  4. Weight is an extraneous variable.
  5. One response variable is the time to complete the journey.
  6. Age is a possible confounding variable. Explain.
  7. Resting metabolic rate is a possible confounding variable. Explain.

6.9 Exercises

Answers to odd-numbered exercises are available in App. E.

Exercise 6.1 The Giant Mine in Yellowknife, Canada, ceased operation in 1999 after \(50\) years, during which \(237\ 000\) tonnes of arsenic trioxide was released. One study (Houben et al. 2016) examined the arsenic concentration in \(25\) lakes within a \(25\) km radius of the mine (\(11\) years after the mine closed), to determine if the arsenic concentration was related to the distance of the lake from the mine. They also recorded:

  • the type of bedrock (volcanic; sedimentary; grandiorite);
  • the ecology type (lowland; upland);

 

  • the elevation of the lake (in metres);
  • the lake area (in hectares); and
  • the catchment area (in hectares).

Use this information to answer the following.

  1. What is the response variable?
  2. What is the explanatory variable?
  3. Is the variable 'Catchment area' likely to be a lurking variable?
  4. Is the variable 'Type of bedrock' likely to be a confounding variable?
  5. What is the best description of the variable 'Ecology type'?
  6. What type of study is this?

Exercise 6.2 A study examined the relationship between diet quality and depression in Australian adolescents (Jacka et al. 2010). The researchers used a sample of \(7114\) adolescents aged \(10\)--\(14\) years old, and also measured information about (p. 435):

...age, gender, socioeconomic status, parental education, parental work status, family conflict, poor family management, dieting behaviours, body mass index, physical activity, and smoking...

  1. Identify the response and explanatory variables.
  2. Which of the other listed variable reasonably could be considered extraneous variables, confounding variables and lurking variables?

Exercise 6.3 A newspaper article (Anonymous 2012) reported that 'Women who drank green tea at least three times a week were \(14\) per cent less likely to develop a cancer of the digestive system'. However, the final paragraph of the article notes that:

Nobody can say whether green tea itself is the reason, since green tea lovers are often more health-conscious in general.

Identify the explanatory and response variables, and explain that final sentence using language introduced in this chapter.

Exercise 6.4 A study recorded the lung capacity (using Forced Expiratory Volume, or FEV, in litres) of children aged \(3\) to \(19\) (Tager et al. 1979; Kahn 2005), and also recorded whether not the children were smokers. One finding was that children who smoke have a larger average FEV (i.e., larger average lung capacity) than children who do not smoke, in general.

Name a confounding variable that may explain this surprising finding. Would it be likely that this variable is a lurking variable?

Exercise 6.5 Consider a study to determine if the percentage of children who consume Ready-To-Eat-Cereals (RTEC) for breakfast is the same for children aged between \(5\) and \(10\), as for children aged between \(11\) and \(15\). The researchers also measured the age of the child, the number of siblings living with the child, and the sex of the child.

  1. Which of these variables are extraneous variables?
    • The sex of the child
    • The percentage consuming RTEC
    • Whether or not the child consumes RTEC
    • The age group of the child
    • The age of the child
    • The number of siblings living with the child
  2. Is the variable 'the sex of the child' a lurking variable?
  3. Is it reasonable to consider the weight of the child as a lurking variable?