9 Study design limitations

So far, you have learnt to ask a RQ, identify different ways of obtaining data, and design the study.

In this chapter, you will learn to identify and describe the limitations of a study. You will learn to:

  • identify when and how study results are internally valid.
  • identify when and how study results are externally valid.
  • identify when and how study results are ecologically valid.

9.1 Introduction

The type of study and how that study is designed can determine how the results of the study should be interpreted. Ideally, a study would be perfectly externally and internally valid, but in practice this is very difficult to achieve. Practically every study has limitations.

The results of a study should be interpreted in light of these limitations: The results should be discussed in light of what the study actually tells, and what it doesn't actually tell us.

Limitations can often be discussed through three components:

  • External validity (the generalisability of the study results outside the sample): The generalisability of the results (Sect. 9.2) to the intended population.
  • Internal validity (the effectiveness of the study in the sample): The steps taken to maximise the internal validity of the study, and the impacts of these on the interpretation of results (Sect. 9.3).
  • Ecological validity (the practicality of the results to real life): The practicality the results in the real world (Sect. 9.4); how the study methods, materials and context approximate the real situation being studied.

All these issues should be considered when considering the study limitations.

Almost every study has limitations. Identifying these limitation, and discussing the impact that they have on the interpretation of the study results, is important and ethical.

Example 9.1 (Interpretation) Smoking was once considered healthy and beneficial: some advertisements used doctors to promote cigarette smoking, and others promoted the benefits of smoking to athletes.232

An advertisement for Camel cigarettes once claimed that 'More doctors smoke Camels than any other cigarette' based on a survey of 113 597 doctors. That certainly is a large sample... but understanding how the data were collected is important.

In fact, the company that owned Camel cigarettes (RJ Reynolds Tobacco Company) conducted the survey (which raises suspicions immediately). Even worse, RJ Reynolds staff interviewed doctors and asked about their smoking habits after RJ Reynolds provided free cartons of Camel cigarettes to the doctors. No wonder more doctors smoked Camel brand!

Concluding that 'More doctors smoke Camels than any other cigarette' in light of this may be technically coprrect, but is highly unethical.

9.2 Limitations: External validity

External validity refers to the ability to generalise the results to other groups in the population apart from the sample studied (Sect. 3.6). For a study to be externally valid, it must first be internally valid.

External validity refers to how well the sample is likely to represent the target population as given in the RQ.

For example, suppose the RQ is 'Among Queenslanders, what proportion own a smart speaker?'.

The study is externally valid if the sample is representative of Queenslanders, and hence the results from the sample are likely to apply to Queenslanders as a whole.

The results do not have to apply to people in the rest of Australia. The intended population, as given in the RQ, is Queenslanders.

External validity refers to the applicability or the generalisability of the results to the target (or intended) population (Example 5.11), which depends on how the sample was obtained. Results from random samples are likely to generalise to the population and be externally valid when appropriately analysed. (The analyses in this book assume all samples are simple random samples.)

Furthermore, results from approximately representative samples may generalise to the population and be externally valid if those in the study are not obviously different than those not in the study.

Example 9.2 (External validity) A New Zealand study233 identified (for well-documented reasons) a particular group to study: 'women of South Asian origin living in New Zealand' (p. 21).

The women in the sample came from a study

... which investigated the health and lifestyle of women of South Asian origin living in New Zealand. Subjects [...] were recruited using a convenience sample method throughout Auckland, which is New Zealand's largest city, and the city in which most South Asian immigrants settle...

--- Gammon et al.234, p. 21

The results may not generalise to the intended population of 'women of South Asian origin living in New Zealand' because all the women in the sample came from only one city in New Zealand (Auckland), and the sample was not a random sample.

The results will not generalise to all New Zealand women, but this is not a limitation: the target population was only 'women of South Asian origin living in New Zealand'. The researchers did not intend the results to apply to all New Zealand women.

Example 9.3 (Using biochar) A study of using biochar235 to grow ginger only used one farm at Mt Mellum, Australia.

While the results may only apply to growing ginger at Mt Mellum, the encouraging results suggest that a wider, more general, study of the impact of using biochar to grow ginger would be worthwhile.

In addition, ginger is usually grown is similar types of climates and soils, so the results may apply to other ginger farms also.

9.3 Limitations: Internal validity

Internal validity refers to how reasonable and logical the results from the study are: the strength of the inferences that can be made from the sample (Sect. 3.6). That is, an internally valid study is effective in demonstrating that the conclusions made from the sample cannot be explained any other way.

Internal validity can be compromised by confounding, the carryover effect, the Hawthorne effect, observer bias, and/or the placebo effect. Consequently, if any of these issues are likely to compromise internal validity, the implications on the interpretation of the results should be discussed.

For example, if the study design implies that the Hawthorne effect is likely to be an issue (since the participants were not blinded), this should be clearly stated, and the conclusion should indicate that the individuals in the study may have behaved differently than usual because (for example) they knew the were in a study.

The internal validity of observational studies is often compromised because confounding can be less effectively managed than for experimental studies.

The internal validity of experimental studies involving people is often compromised because people must be informed that they are participating in a study.

Example 9.4 (Internal validity) In a study of the hand-hygiene practices of paramedics,236 self-reported hand-hygiene practices were very different than what was reported by peers:

...social desirability and identity may have led to the intentional misreporting of IPC [infection prevention and control] behaviors in favor of better compliance by the participants. Evidence for this is that participants reported much higher levels of compliance for themselves than their colleagues

--- Barr et al.237, p. 777.

That is, when participants knew they were being studied, they may have given responses that made their own behaviours appear better than their colleagues. This is a study limitation that was necessary to discuss.

Example 9.5 (Internal validity) A study of using 'vibration training' on elderly men listed some limitations of their study:

The potential for observer bias is also acknowledged, as the researcher was not blinded and had prior knowledge of the research aims, disease status, and intervention. As such, these could all have influenced data recording. The researcher tried to minimize the risk of bias by following a standardized protocol for enrolment. The potential of reporting bias and observer bias could be reduced by implementing blinding in future studies.

--- Anna Kabata-Piżuch et al.238, p. 10

A study239 examined the food choices made when subjects were asked to shop for ingredients to make a last-minute meal.

Half were told to prepare a 'healthy meal', and the other half told just to prepare a 'meal'. Part of the Discussion stated:

Another limitation is that results report findings from a simulated purchase. As participants did not have to pay for their selection, actual choices could be different. Participants may also have not behaved in their usual manner since they were taking part in a research study, a situation known as the Hawthorne effect.

--- Botelho et al.240, p. 436

9.4 Limitations: Ecological validity

The practicality of the study results in the real world should also be discussed. This is called ecological validity.

Definition 9.1 (Ecological validity) A study is ecologically valid if the study methods, materials and context approximate the real situation being studied.

Studies don't need to be ecologically valid to be useful; much can be learnt under special conditions, as long as the potential limitations are understood when applying the results to the real world. Although ecological validity is not essential for a good study, ecological validity is useful if it is possible to achieve.

The ecological validity of experimental studies may be compromised because the experimental conditions are sometimes contrived.

Example 9.6 (Ecological validity) Consider a study to determine how likely it is that people will buy a coffee in a reuseable cup.

We could ask people about their intentions. This study may not be ecologically valid, as how people act in the real world may not align with what they say, especially when social pressures exist to use reusable cups.

An alternative study involves watching people buy coffees at various coffee shops, and record what people actually do in practice.

This second study is more likely to be ecologically valid, as we are watching actual behaviour in the real world.

A study was completed to observe the effect of using high-mounted rear brake lights,241 which are now commonplace. The American study showed that such lights reduced rear-end collisions by about 50%. However, after making these lights mandatory, rear-end collisions reduced by only 5%. Why?

9.5 Summary

The limitations in a study need to be identified. Limitations may be related to:

  • internal validity (effectiveness);
  • external validity (generalisability); and/or
  • ecological validity (practicality).

9.6 Quick review questions

A study242 examined the effect of peer pressure from passengers among teenage male drivers; the aim was to

...experimentally test the effects of passenger presence and social influence [...] of male adolescent novices in a simulated driving task.

Bingham et al.243, p. 126

The use of a driving simulator was justified as:

... driving simulation has been shown to be an externally valid predictor of real-world driving

Bingham et al.244, p. 125

The Discussion section of the article includes a subsection called 'Strengths and limitations'. Part of that sub-section reads:

... participants were closely clustered around average rates of resistance to peer influence for this age group [...] so it is unclear to what extent these findings would generalize to participants with weaker resistance to peer influences.

Bingham et al.245, p. 135

Later, the paper reports:

... the use of an age-peer [passenger assigned by the researchers] allowed substantial experimental control, it may have provided participants with an artificial experience compared to the influence of actual friends.

Bingham et al.246, p. 135

  1. The study used \(n = 52\) 16- and 17-year-old males in the study. Should the external validity of the study be criticised for only using teenage males in the study, and not teenage females?
  2. What does the second last quotation above mean?
  3. True or false: The Hawthorne effect is likely to be an issue in this study.

  1. Which of the following are true?

    • When interpreting the results of studies, the steps taken to maximize internal validity should be considered.
    • If studies are not externally valid, then they are not very useful.
    • When interpreting the results of studies, the steps taken to maximize external validity do not need to be considered.
    • When interpreting the results of studies, ecological validity is about the impact of the study on the environment.

When interpreting the results of studies, we consider the practicality ( validity), the generalizability ( validity) and the effectiveness ( validity).

Internal validity refers to issues such as and the Hawthorne effect.

External validity refers to methods.


9.7 Exercises

Selected answers are available in Sect. D.9.

Exercise 9.1 A student project at the university where I work had the RQ:

Among UniSC students on-campus, is the percentage of word retention higher in male students than female students?

When they were discussing external validity, they said:

We cannot say whether or not that the general public have better or worse word retention compared to the students that we will be studying.

Why is the statement not relevant?

Exercise 9.2 Despite their common use, no experimental scientific evidence shows that parachutes are effective.247 To obtain evidence, researchers studied this scenario.248 Part of the Abstract for the paper (slightly edited for clarity) says:

Objective To determine if using a parachute prevents death or major traumatic injury when jumping from an aircraft.

Design Randomized controlled trial.

Setting Private or commercial aircraft between September 2017 and August 2018.

Participants 92 aircraft passengers aged 18 and over were screened for participation. 23 agreed to be enrolled and were randomized.

Intervention Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded).

Main outcome measures Composite of death or major traumatic injury (defined by an Injury Severity Score over 15) upon impact with the ground measured immediately after landing.

Results Parachute use did not significantly reduce death or major injury (0% for parachute v 0% for control; \(P>0.9\)).

Conclusions Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention. However, the trial was only able to enroll participants on small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps [...]

--- Yeh et al.249

Based on this information:

  1. Carefully define POCI.
  2. What type of study is this: observational or experimental?
  3. What are the variables?
  4. Comment on the ecological validity of this study.
  5. Comment on the limitations of the study.
  6. What are the conclusions?

Exercise 9.3 A study of how well hospital patients sleep at night250 set out to

...to investigate the perceived duration and quality of patient sleep and identify any environmental factors associated with patient-reported poor sleep in hospital.

--- Delaney et al.251 p. 1

In discussing the study, the researchers state:

Patients and nursing staff were recruited for this study. Non-probability convenience sampling was used to recruit patients to participate...

--- Delaney et al.252 p. 2

Later, while discussing the limitations, the researchers state:

While the multiple methods of data collection and inclusion of 15 clinical areas are strengths of this study, the results may not be generalisable to all hospitals or all ward areas [...] while most healthy individuals sleep primarily or exclusively at night, it is important to consider that patients requiring hospitalization will likely require some daytime nap periods. This study looks at sleep only in the night-time period 22:00--07:00h, without the context of daytime sleep considered.

-- Delaney et al.253 p. 7

Discuss these issues using the language introduced in this chapter.