9.3 Limitations: Internal validity
Internal validity refers to how reasonable and logical the results from the study are: the strength of the inferences that can be made from the sample (Sect. 3.6). That is, an internally valid study is effective in demonstrating that the conclusions made from the sample cannot be explained any other way.
Internal validity can be compromised by confounding, the carryover effect, the Hawthorne effect, the observer effect, and/or the placebo effect. Consequently, if any of these issues are likely to compromise internal validity, the implications on the interpretation of the results should be discussed.
For example, if the study design implies that the Hawthorne effect is likely to be an issue (since the participants were not blinded), this should be clearly stated, and the conclusion should indicate that the individuals in the study may have behaved differently than usual because (for example) they knew the were in a study.
The internal validity of observational studies is often compromised because confounding can be less effectively managed than for experimental studies.
Example 3.5 (Internal validity) In a study of the hand-hygiene practices of paramedics (Barr et al. 2017), self-reported hand-hygiene practices were very different than what was reported by peers:
That is, when participants knew they were being studied, they may have given responses that made their own behaviours appear better than their colleagues. This is a study limitation that was necessary to discuss.…social desirability and identity may have led to the intentional misreporting of IPC [infection prevention and control] behaviors in favor of better compliance by the participants. Evidence for this is that participants reported much higher levels of compliance for themselves than their colleagues
— Barr et al. (2017), p. 777.
A study (Botelho et al. 2019) examined the food choices made when subjects were asked to shop for ingredients to make a last-minute meal.
Half were told to prepare a ‘healthy meal,’ and the other half told just to prepare a ‘meal.’ Part of the Discission stated:
Another limitation is that results report findings from a simulated purchase. As participants did not have to pay for their selection, actual choices could be different. Participants may also have not behaved in their usual manner since they were taking part in a research study, a situation known as the Hawthorne effect.
— Botelho et al. (2019), p. 436