9.1 Introduction
The type of study and how that study is designed can determine how the results of the study should be interpreted. Ideally, a study would be perfectly externally and internally valid, but in practice this is very difficult to achieve. Practically every study has limitations.
The results of a study should be interpreted in light of these limitations: The results should be discussed in light of what the study actually tells, and what it doesn’t actually tell us.
Limitations can often be discussed through three components:
- External validity (the applicability of the study results outside the sample): The generalisability of the results (Sect. 9.2) to the intended population.
- Internal validity (the effectiveness of the study in the sample): The steps taken to maximise the internal validity of the study, and the impacts of these on the interpretation of results (Sect. 9.3).
- Ecological validity (the practicality of the results to real life): The practicality the results in the real world (Sect. 9.4); how the study methods, materials and context approximate the real situation being studied.
All these issues should be considered when considering the study limitations.
Example 9.1 (Interpretation) Smoking was once considered healthy and beneficial: some advertisements used doctors to promote cigarette smoking, and others promoted the benefits of smoking to athletes (Ingalls 1936).
An advertisement for Camel cigarettes once claimed that ‘More doctors smoke Camels than any other cigarette’ based on a survey of 113 597 doctors. That certainly is a large sample… but understanding how the data were collected is important.
In fact, the company that owned Camel cigarettes (RJ Reynolds Tobacco Company) conducted the survey (which raises suspicions immediately). Even worse, RJ Reynolds staff interviewed doctors and asked about their smoking habits after RJ Reynolds provided free cartons of Camel cigarettes to the doctors. No wonder more doctors smoked Camel brand!
Concluding that ‘More doctors smoke Camels than any other cigarette’ in light of this is highly unethical.