1.9 Design issues: Overview
In summary, issues to consider when designing a study, when possible, include:
- Minimising confounding (and lurking variables);
- Minimising the carryover effect;
- Minimising the Hawthorne effect;
- Minimising the observer effect;
- Minimising the placebo effect.
Ways to minimize the impact of these have been discussed (Fig. 1.7), but is not always possible. These effects are important to understand, so studies can be designed to manage or minimise their influence (to maximise internal validity). This ensures that the results and conclusions from our studies are correctly interpreted (that is, noting, for example, how the Hawthorne effect may have influenced the conclusions).
Often, however, some (or all) of these issues cannot be well managed. For instance, individuals often know they are involved in an experimental study (Hawthorne effect). In these cases, the impacts should be minimized as far as possible, and then the likely impact that these issues have on our conclusions discussed. The impact of these issues are often reported as limitations in a journal article (Chap. 3), perhaps part of the Discussion section.
Example 1.20 (Study limitations) A study of alcohol use in college females reported these limitations of their study:
The present study has several limitations. First, data were collected over 15 years ago [...] Second, only college females were assessed and findings may not generalize to college males or to broader groups of young adults [...] Third, alcohol and caffeine consumption variables were all self-reported...
--- Kelpin et al. (2018), p. 3
Figure 1.7: Design considerations. Note: Lurking variables become confounding variables when measured, observed, assessed or recorded in the study, and then they can be managed. The arrows mean that the design issue can be partially managed by the indicated means
Example 1.21 (Study design) In a study of student paramedics comparing chest compressions with dominant and non-dominant hands (Cross et al. 2019), as discussed in Example 1.18, the participants were partially blinded: they were blinded to the purpose of the study, but not to which group they were allocated.
The analyst was also blinded to the group allocations.
Later, the article reports that:
This study used a number of good design features....participants were allocated randomly to one of two groups: 'dominant hand on chest' or 'non-dominant hand on chest'. Group allocation was determined by a computer-generated randomisation schedule...
Cross et al. (2019), p. 3