6 Overview of internal validity

So far, you have learnt to ask a RQ, select a study design, and find a sample to study. In this chapter, you will learn how to ensure that the conclusions are logical and sound. You will learn to:

• identify issues that might negatively impact internal validity.
• identify issues that might impact the values of the response variables.
• identify extraneous, confounding and lurking variables.

6.1 Introduction

Consider the letter-typing RQ again (from Example 5.3):

For students in this course this semester, is the average number of words typed per minute on a keyboard the same for females and males?

For this study, the population is 'Students in this course this semester' and the outcome is 'the average number of letters typed per minute' (say, 'typing speed'). The between-individuals comparison is between females and males, and since the sex of the student cannot be manipulated by the researchers, there is no intervention. The study is observational.

The measured the typing speed (the response variable) of many individuals will vary (Fig. 6.1; left bars): It is very unlikely that every student in the study will record the same typing speed. The typing speeds can vary due to many reasons (Fig. 6.2):

• The explanatory variable (Sect. 6.3): The values of the explanatory variable may influence the values of the response variable; of course, they may not either. The purpose of the study is to find if, or to what extent, this is true. In this example, the explanatory variable is the sex of the student.
• Other variables (Sect. 6.4): Other variables (apart from the explanatory variables) may influence the response variable (perhaps more so than the explanatory variable) such as 'age', or 'whether or not the person wears glasses'. The impact of these variables can be understood if the study is designed appropriately.
• Design issues (Sect. 6.5): The way the study is designed can also influence the values of the response variable. These can mean disaster if not managed properly.
• Chance, or randomness (Sect. 6.6): Even the same person doing the same thing repeatedly under the same conditions will not record exactly the same typing speed every attempt. This is unavoidable, but good design can minimize the size of this variation.

The purpose of the study is to explore the relationship between the response variable and the explanatory variable... but other issues can obscure that relationship. An internally valid study is one designed to focus on the relationship between the response and explanatory variables, and eliminating other influences (such as extraneous variables, design issues, and chance) by understanding these as far as possible (Fig. 6.1).

One goal of study design is to maximise internal validity: to isolate the relationship of interest, by eliminating, as far as possible, all other possible explanations.

FIGURE 6.1: The total variation in the response variable appears all as unexplained chance (a). Good design can eliminate some of that variation, and usually we wish to study the relationship with an explanatory variable (c). The unexplained chance variation is actually comprised of several components (d), and the effect of the explanatory variable is difficult to see (relative to chance), unless the effect of design factors and extraneous variables have been elimiated (e)

Internal validity is one of the most important properties of scientific studies, and is important for reasoning about evidence more generally. Designing studies to maximise internal validity is the focus of Chaps. 7 (experimental studies) and 8 (observational studies). This chapter provides an overview.

Data collection is often tedious, time consuming and expensive. You usually get one chance to collect data, but data can be analysed as many times as necessary. Design the study properly the first time!

6.2 Bias

Bias refers to any misrepresentation that can lead to a false conclusion. Bias compromises the results or inferences in a study, which may lead to inaccurate conclusions. Bias can be introduced at any part of the research process.

Definition 6.1 (Bias) Bias refers to anything systematic which may cause differences between sample statistics and population parameters.

Bias may occur during study design (poor internal validity), data collection (selection bias; Sect. 5.6), analysis, or interpreting (Sect. 9) and reporting results. The online Catalogue of Bias list over 60 different ways that studies can be biased, though this book only discusses a small number of possible biases.

FIGURE 6.2: Other factors can influence the values of the response variable, besides the relationship of interest

6.3 Explanatory variable changes

Changes in the values of the explanatory variable may be associated with changes in the values of the response variable. However, it may not; after all, determining if a relationship exists between the response and explanatory variables (or the extent of this relationship) is the purpose of the study. If nothing else influenced the values of the response variable, life would be easy: any change of a given size in the value of the explanatory variable would always result in a change of the same size in the value of the response variable.

Example 6.1 (Explanatory variable) In the typing-speed study (Sect. 6.1), the explanatory variable is the sex of the person. If nothing else influenced typing speed, all females would record the same typing speed every time, and all males would record the same typing speed every time. This is clearly unreasonable.

FIGURE 6.3: Explanatory variables may influence the values of the response variable

6.4 Importance of identifying extraneous variables

Other variables (besides the explanatory variable) almost certainly exist which are associated with changes in the value of the response variable. These are called extraneous variables.

Definition 6.2 (Extranaeous variable) An extraneous variable is any variable that is (potentially) associated with the response variable, but is not the explanatory variable.

Example 6.2 In the typing-speed study (Chap. 6.1), potential extraneous variables may include age, the presence or absence of certain medical conditions, the level of familiarity with computers, etc.

All extraneous variables are, by definition, potentially related to the response variable. They may or may not be associated with the explanatory variable as well. Extraneous variables may have other names too (Table 6.1), though these names are used inconsistently (P. K. Dunn et al. 2016).

When an extraneous variable is also related to the explanatory variable, the extraneous variable is called a confounding variable. A confounding variable can obscure the true relationship between the response and explanatory variables (i.e., confounding variables can bias the results).

Definition 6.3 (Confounding variable) A confounding variable (or a confounder) is an extraneous variable associated with the response and explanatory variables (Fig. 6.4).

Definition 6.4 (Confounding) Confounding is when a third variable influences the observed relationship between the response and explanatory variable.

A relationship between the response and explanatory variables may be apparent, but only because both of these variables are related to the confounding variable (Fig. 6.4).

Example 6.3 (Confounding variables) People who carry cigarette lighters are more likely to get lung cancer. The reason this relationship exists, however, is because of a confounding variable: whether or not the person is a smoker. 'Whether or not the person is a smoker' is probably related to both the response and explanatory variables:

• A smoker is more likely to carry a cigarette lighter (the explanatory variable) than a non-smoker; and
• A smoker is more likely to develop lung cancer (the response variable) than a non-smoker.

Consider this RQ:

Among university students, is the percentage of females who know their own blood pressure the same as the percentage of males who know their own blood pressure?

For this RQ, the explanatory variable is the sex of person, and the response variable is whether a student knows their own blood pressure. A potential confounding variable is 'The program of study', since this is (potentially) related to both the response and explanatory variables:

• 'Program of study' is related to sex (the explanatory variable): a higher percentage of females study nursing, while a greater percentage of males study engineering (at least, in Australia).
• 'Program of study' is related to knowing your blood pressure (the response variable): nursing students probably practice taking each others blood pressures so probably know theirs, whereas engineering students do not.

Managing confounding is very important, as confounding can completely change the relationship between the response and explanatory variables (see Sect. 14.3) and hence can compromise internal validity. Managing confounding is discussed in Sects. 7.2 and 8.2.

FIGURE 6.4: Confounding variables are extraneous variables associated with the response and explanatory variables

If the values of potential confounding variables are recorded, their impact can be managed. However, sometimes the values of the confounding variables are not recorded; then, they are called lurking variables (Fig. 6.5). Failure to acknowledge lurking variables can lead to wrong conclusions (for example, see Sect. 14.3).

Definition 6.5 (Lurking variable) A lurking variable is an extraneous variable associated with the response and explanatory variables (that is, is a confounding variable), but whose values are not recorded in the study data.

FIGURE 6.5: Lurking variables are associated with the response and explanatory variables, but are not recorded

Example 6.4 (Lurking variables) A study reported by Joiner (1981) and Wilson Jr (1952) examined the strength of plastic parts. The aim was to determine if the time in the production mould influenced the strength of the part. In the study,

Hot plastic was introduced in the mold, pressed for 10 seconds, and removed. Another batch was then introduced into the same mold, pressed for 20 seconds, and so on, the time increasing with each batch.

--- Wilson Jr (1952), p. 55--56

Longer mould times (the explanatory variable) were found to be associated with greater plastic strength (the response variable).

However, further study revealed a lurking variable: the mould temperature. This was a lurking variable since:

• Higher mould temperatures (the lurking variable) were associated with greater strength (the response variable); and
• Higher mould temperatures (the lurking variable) were experienced by later batches with longer mould times (the explanatory variable), since the mould was hotter for the later batches.

The cause of the greater strength was not the time in the mould; it was the higher temperature experienced by the later moulds (Fig. 6.6).

FIGURE 6.6: An example of a lurking variable. Left: The relationship as originally understood; right: the relationship after the lurking variable was eventually exposed

To clarify (Table 6.1):

• All extraneous variables are related to the response variable, by definition.
• Some extraneous variables are also called confounding variables, if they are also related to the explanatory variable.
• Some confounding variables are also called lurking variables, if they are not recorded.

Unknown extraneous variables become part of variation due to chance (i.e., unexplained). These terms are not always used consistently by all researchers (Flanagan-Hyde 2005), but the ideas are important nonetheless.

TABLE 6.1: The relationship between the population and the individuals

Type	Associated with response	Associated with response and explanatory
Measured or observed	No special name: extraneous	Confounding (not lurking)
Not measured or observed	Becomes part of 'chance'	Lurking

To avoid lurking variables, researchers generally collect lots of information about the individuals in the study (such as age and sex if the study involves people) and circumstances of the individuals in the study (such as the temperature at the time of data collection) that may be relevant, in case they are confounding variables.

Can you think of possible extraneous variables in the letter-typing study (Sect. 6.1)?

6.5 Importance of study design

Many aspects of the study design can influence the observed relationship between the response and explanatory variable (i.e., can bias the results). Good design principles can minimise other impacts, so the focus remains on the influence of the explanatory variable on the response variable (Fig. 6.1).

Since many aspects of the study design may be under the control of the researchers, study design is very important for reducing bias and improving internal validity. The study design principles are discussed at length soon (Chaps. 7 and 8).

Example 6.5 (Design) The typing-speed study (Sect. 6.1) could be poorly designed. Suppose females were always asked to use their dominant hand, and males always asked to use their non-dominant hand. Females would probably have a faster average time, simply because they use their dominant hands.

6.6 Chance and variation in the response

Chance (or natural) variation refers to variation that cannot otherwise be explained: even repeating a study exactly the same way every time on the same individuals will not always produce the same values of the response variable.

Chance variation makes the influence of the explanatory variable (which we want to study) hard to detect, so minimising chance variation is important (Fig. 6.1). Minimising the amount of the chance variation requires using good design principles, and measuring as many other extraneous variables that may explain variation in the response variable as is reasonable.

Chance can impact the values of the response variable in different ways:

• Each individual can produce different values of the response variable each time the response variable is measured (within-individuals variation); and
• Each individual in the study can produce different values of the response variable compared to other individuals (between-individuals variation):

We need different strategies to understand each of these sources of variation:

• To estimate the amount of variation within individuals: many observations are needed from each unit of analysis (individual).
• To estimate amount of variation between individuals: many units of analysis (individuals) are needed.

Since between-individual variation is usually more variable (i.e., larger variation) than the within-individual variation, using many individuals is usually more important than using a smaller number of individuals many times each.

Consider the typing-speed study (Sect. 6.1) again. What are the advantages and disadvantage of measuring the typing speed for:

1. one female 30 times?
2. 30 different females once each?
3. 10 different females, three times each?

1. We would learn a lot about that female... but very little about females in general.
2. We would learn a lot about females in general... but have one measurement from each. Since we might expect that the same person might produce similar (not necessarily identical) typing speed, this is not really a problem.
3. We would learn a lot about females in general... and a little about each female too.

6.7 Summary

In a research study, the main interest is usually the relationship between a response variable and explanatory variable. However the values of the response variable can be influenced by things other than the explanatory variable: extraneous variable (other variables that aren't of primary interest), the study design, and chance.

Some extraneous variables are also related to the explanatory variable, and are called confounding variables (and are lurking variables if they are not recorded. If the study design makes it difficult to separate the relationship between the response and explanatory variable from other possible causes, the study has poor internal validity.

6.8 Quick review questions

The Giant Mine in Yellowknife, Canada, ceased operation in 1999 after operating for 50 years, during which 237,000 tonnes of arsenic trioxide was released. One study (Houben et al. 2016) examined the arsenic concentration in lake water from 25 lakes within a 25km radius of the mine (11 years after the mine closed), to determine if the arsenic concentration was related to the distance of the lake from the mine. They also recorded:

• the type of bedrock (volcanic; sedimentary; grandiorite);
• the ecology type (lowland; upland);
• the elevation of the lake (in metres);
• the lake area (in hectares); and
• the catchment area (in hectares).

Use this information to answer the following.

1. What is the response variable?

   Arsenic concentrationDistance of the lake from the mineType of bedrockEcology typeElevationLake areaCatchment area

2. What is the explanatory variable?
   

   Arsenic concentrationDistance of the lake from the mineType of bedrockEcology typeElevationLake areaCatchment area

3. Is the variable "Catchment area" likely to be a lurking variable?
   

   No: It is not related to the response variableNo: It is measured and recorded, so cannot be a lurking variableYes: It may be related to the response and explanatory variablesYes: It may be related to the response variable

4. Is the variable "Type of bedrock" likely to be a confounding variable?
   

   No: It is not related to the response variableNo: It is measured and recorded, so cannot be a confounding variableYes: It may be related to the response and explanatory variablesYes: It may be related to the response variable

5. What is the best description of the variable "Ecology type"?
   

   Response variableExplanatory variableConfounding variableLurking variable

6. What type of study is this?
   

   Experimental, because this is a scientific study involving chemical analysis of water samplesExperimental, because the researchers need to do some laboratory work to determine the arsenic concentrationsObservational, because the researchers do not determine the distance of the lakes from the mine siteObservational, because the researchers are taking observations about the lakes (such as elevation and bedrock type)

7. Which of the following can be used to manage confounding in experiments?

   • Blinding the individuals Yes No
   • Using a control group Yes No
   • Using special methods of analysis Yes No
   • Randomly allocating treatments to groups Yes No
   • Blinding the researchers Yes No

8. Which of the following statements are true?

   • Experimental studies must use random samples. Yes No
   • An experimental study must blind the researchers. Yes No
   • An experimental study must blind the participants. Yes No
   • Experimental studies must use a control group. Yes No
   • In experimental studies, the treatments must be allocated by the researchers. Yes No

6.9 Exercises

Selected answers are available in Sect. D.6.

Exercise 6.1 A study examined the relationship between diet quality and depression in Australian adolescents (Jacka et al. 2010). The researchers used a sample of 7114 adolescents aged 10--14 years old, and also measured information about:

...age, gender, socioeconomic status, parental education, parental work status, family conflict, poor family management, dieting behaviours, body mass index, physical activity, and smoking...

--- Jacka et al. (2010), p. 435

1. Identify the response and explanatory variables.
2. Which of the other listed variable reasonably could be considered extraneous variables, confounding variables and lurking variables?

Exercise 6.2 A newspaper article (Anonymous 2012) reported that "Women who drank green tea at least three times a week were 14 per cent less likely to develop a cancer of the digestive system". However, the final paragraph of the article notes that:

Nobody can say whether green tea itself is the reason, since green tea lovers are often more health-conscious in general.

Identify the explanatory and response variables, and explain that final sentence using language introduced in this chapter.

Exercise 6.3 A study recorded the lung capacity (using Forced Expiratory Volume, or FEV, in litres) of children aged 3 to 19 (Tager et al. 1979; Kahn 2005), and also recorded whether not the children were smokers. One finding (P. K. Dunn and Smyth 2018) was that children who smoke have a larger average FEV (i.e., larger average lung capacity) than children who do not smoke.

Name a confounding variable that may explain this surprising finding. Would it be likely that this variable is a lurking variable?

Exercise 6.4 Consider a study to determine if the percentage of children who consume Ready-To-Eat-Cereals (RTEC) for breakfast is the same for children aged between 5 and 10, as for children aged between 11 and 15. The researchers also measured the age of the child, the number of siblings living with the child, and the sex of the child.

1. Which variables are extraneous variables?
   • The sex of the child Yes No
   • The percentage consuming RTEC Yes No
   • Whether or not the child consumes RTEC Yes No
   • The age group of the child Yes No
   • The age of the child Yes No
   • The number of siblings living with the child Yes No
2. Is the variable "the sex of the child" a lurking variable?
   Yes: It is likely to be related to the response variableNo: It is the explanatory variableNo: It is a variable that is recorded so it can't be a lurking variableYes: it is likely to be related to the explanatory variableYes: It is likely to be related to the response and explanatory variable
3. Is it reasonable for the weight of the child to be a lurking variable?
   No: it can't be a lurking variable because it is probably not related to all the extraneous variables.Yes, because it is related to whether or not the child consumes RTEC.Yes: It is not measured, and could be related to the age of the child, and whether or not the child consumes RTEC.No: It can't be a variable because it is not measured.Yes, because it is probably related to age group of the child.

---

Peter K. Dunn, 2023: CC BY-NC-SA 4.0