3 Types of study designs

So far, you have learnt how to ask a RQ.

In this chapter, you will study the details of how to design a method for collecting the data needed to answer the RQ. You will learn to:

  • design scientifically sound studies to answer simple quantitative research questions.
  • design ethical studies.
  • describe the various types of quantitative research studies.
  • compare and contrast experimental and observational studies.
  • describe and identify retrospective, prospective and cross-sectional observational studies.
  • describe and identify true experimental and quasi-experimental studies.
  • explain external and internal validity.

3.1 Three types of study designs

From the RQ, we know what data must be collected from the individuals in the study (the response and explanatory variables)... but how do we design a study to obtain this data? After all, data are important: they are the means by which the RQ is answered.

Three broad methods for obtaining data are to use:

  • Descriptive studies (Sect. 3.2), for answering Descriptive RQs;
  • Observational studies (Sect. 3.3), for answering Relational RQs; or
  • Experimental studies (Sect. 3.4), for answering Interventional RQs.

The type of study depends on the type of RQ.

Example 3.1 (Research design) Suppose we wish to compare the effects of echinacea on the symptoms of the common cold (based on Barrett et al.90).

How could we design such a study to collect the necessary data?

What decisions would you need to make?

3.2 Descriptive studies

Descriptive studies are used to answer descriptive RQs (Fig. 3.1).

Definition 3.1 (Descriptive study) In a descriptive study, researchers only focus on collecting, measuring, assessing or describing an outcome in the population.

A descriptive study, used to answer a descriptive RQ

FIGURE 3.1: A descriptive study, used to answer a descriptive RQ

Example 3.2 (Descriptive study) Consider this RQ:

For overweight men over 60, what is the average increase in heart rate after walking 400 metres?

The outcome is the average increase in heart rate. The response variable is the increase in heart rate for the individual men.

The increase in heart rate would need to be found by measuring each man's heart rate before the walk, then their heart rate after the walk, and finding the difference between them. The increase in heart rate would be computed as the after heart rate minus the before heart rate.

Some of these differences might be positive numbers (heart rate went up), and some may be negative numbers (heart rate went down).

No comparison being made: every man in the study is treated in the same way. This is a descriptive RQ, which can be answered by a descriptive study.

3.3 Observational studies

Observational studies (Fig. 3.2) are used to answer relational RQs. They are a commonly-used study design, and sometimes are the only possible study design that can be used.

Definition 3.2 (Observational study) In an observational study, researchers do not impose, and cannot manipulate, the comparison or connection upon those in the study to (potentially) change the response of the participants.

An observational study, used to answer a relational RQ

FIGURE 3.2: An observational study, used to answer a relational RQ

Definition 3.3 (Condition) Conditions: The conditions of interest that those in the observational study are exposed to.

Example 3.3 (Observational study) Consider again this RQ:91

Among Australian teens with a common cold, is the average duration of cold symptoms shorter for teens taking a daily dose of echinacea compared to teens taking no medication?

This would be a relational RQ if the researchers do not impose the echinacea (that is, the individuals make this decision themselves). For this RQ, the conditions would be taking echinacea, or not taking echinacea (Fig. 3.3).

Observational studies

FIGURE 3.3: Observational studies

Broadly speaking, three types of observational studies exist (Table 3.1):

TABLE 3.1: The three types of observational studies
Type O C Reference
Retrospective Now Earlier Sect. 3.3.1
Prospective Later Now Sect. 3.3.2
Cross-sectional Now Now Sect. 3.3.3

These differ in when the response and explanatory variables are observed. Many specific types of observational studies exist (case-control studies; cohort studies; etc.), but we will not delve into these.

3.3.1 Retrospective studies

In retrospective studies, the response variable is observed now, and the researchers look back to see the value of the explanatory variable in the past (e.g., case-control studies).

Example 3.4 (Retrospective studies) An Australian study92 examined patients with and without sporadic motor neurone disease (SMND), and asked about past exposure to metals.

The response variable (whether or not the respondent had SMND) is assessed now, and whether or not they had exposure to metals (explanatory variable) is assessed from the past. This is a retrospective observational study.

3.3.2 Prospective studies

In prospective studies, the explanatory variable is determined now, and researchers look ahead to assess or measure the response variable (e.g., prospective cohort studies).

Example 3.5 (Prospective studies) A study93 measured the softdrink consumption of men, and determined who experienced gout over the following 12 years.

The response (whether or not the individuals experience gout) is determined in the future. The explanatory variable (the amount of softdrink consumed) is measured now. This is a prospective observational study.

3.3.3 Cross-sectional studies

In cross-sectional studies, both the response and explanatory variables are gathered now.

Example 3.6 (Cross-sectional studies) A study94 asked older Australian their opinions of their own food security, and recorded their living arrangements.

Individuals' responses to both both the response variable and explanatory variable are gathered now. This is a cross-sectional observational study.

In South Australia in 1988--1989, 25 cases of legionella infections (an unusually high number) were investigated.95 All 25 cases were gardeners, with hanging baskets of ferns.

Researchers compared 25 cases with legionella infections with 75 non-cases, matching on the basis of age (within 5 years), sex, post codes. The use of potting mix in the previous four weeks was associated with an increase in the risk of contracting illness of about 4.7 times.

What type of observational study is this?

Retrospective: people were identified with an infection, and then the researchers looked back at past activities.

3.4 Experimental studies

Experimental studies (Fig. 3.4), or experiments, are commonly-used study designs. Well-designed experimental studies can establish a cause-and-effect relationship between the response and explantory variables. However, using experimental studies is not always possible.

Experimental studies have an intervention, and so experimental studies are used to answer interventional RQs. The researchers impose and can manipulate the values of the explanatory variable: they create changes in the explanatory variable, and record the changes in the response variable.

Definition 3.4 (Experiment) An experimental study (or an experiment), has an intervention: the researchers impose and can manipulate the values of the explanatory variable.

The researchers allocate treatments (i.e., apply the intervention).

An experimental study, used to answer interventional RQs

FIGURE 3.4: An experimental study, used to answer interventional RQs

Definition 3.5 (Treatments) Treatments are the conditions of interest that those in the study can be exposed to (as the explanatory variable). In experiments, treatments are imposed by researchers.

Two types of experimental studies (Table 3.2) are:

TABLE 3.2: Comparing experimental designs (descriptive studies do not have any comparison or connection groups)
Study type Do researchers allocate who or what to groups? Do researchers allocate treatments to groups? Reference
True experiment Yes Yes Sect. 3.4.1
Quasi-experiment No Yes Sect. 3.4.2
Observational No No Sect. 3.3

3.4.1 True experimental studies

True experiments are commonly used, but conducting a true experiment is not always possible. An example of a true experiment is a randomised controlled trial, often used in drug trials.

Definition 3.6 (True experiment) In a true experiment, the researchers:

  1. allocate treatments to groups of individuals (i.e., allocate the values of the explanatory for the individuals), and
  2. determine who or what individuals are in those groups.

While these may not actually happen explicitly, they can happen conceptually.

Example 3.7 (True experiment) The echinacea study96 (Sect. 2.7) could be designed as a true experiment. The researchers would allocate individuals to one of two groups, and then decide which group took echinacea and which group did not (Fig. 3.5).

True experimental studies

FIGURE 3.5: True experimental studies

A researcher wants to examine the effect of an alcohol awareness program97 on the amount of alcohol consumed in O-Week. She runs the program at UQ only, then compared the average amount of drinking per person at two universities (A and B). What type of study is this: observational or true experimental? Answer these questions to help:

  1. Does the researcher allocate treatments to the groups?
  2. Does the researcher allocate subjects to groups?

It is neither.

The researcher did not determine the groups: the students (not the researcher) would have chosen University A or University B for many reasons. The researcher did decide how to allocate the program to University A or University B.

3.4.2 Quasi-experimental studies

Quasi-experiments are similar to true experiments, but treatments are allocated to groups that already exist.

Definition 3.7 (Quasi-experiment) In a quasi-experiment, the researchers:

  • allocate treatments to groups of individuals (i.e., allocate the values of the explanatory variable to the individuals), but
  • do not determine who or what individuals are in those groups.

Example 3.8 (Quasi-experiments) The echinacea study (based on Barrett et al.98) (Sect. 2.7) could be designed as a quasi-experiment. The researchers would need to find (not create) two existing groups of people (say, from two different suburbs) then decide which group took echinacea and which group did not (Fig. 3.6).

Quasi-experimental studies

FIGURE 3.6: Quasi-experimental studies

3.5 Comparing study designs

In experimental studies, researchers create differences in the explanatory variable through allocation, and note the effect this has on the response variable. In observational studies, researchers observe differences in the explanatory variable, and observe the values in the response variable. Different RQs require different study designs (Table 3.3).

TABLE 3.3: Study types and research questions
RQ type P O C I Study type
Descriptive Yes Yes Descriptive
Relational Yes Yes Yes Observational
Interventional Yes Yes Yes Yes Experimental

Importantly, only well-designed true experiments can show cause-and-effect. In general, well-designed true experiments provide stronger evidence than quasi-experiments, which produce stronger evidence than observational studies.

Example 3.9 (Cause and effect) Many studies have reported that the bacteria living in the gut of people on the autism spectrum is different than the bacteria in the gut of people not on the autism spectrum (Dae-Wook Kang et al.99, Lucius Kang Hua Ho et al.100).

However, these studies have been observational, so the suggestion of a cause-and-effect relationship may be inaccurate.

Other studies101 propose that the relationship works the other way: people on the autism spectrum are more likely to be "picky eaters", which contributes to the differences in their gut bacteria.

Although only experimental studies can show cause-and-effect, experimental studies are often not possible for ethical, financial, practical or logistical reasons. The animation below compares observational, quasi-experimental and true experimental designs. In addition, experimental studies may suffer from lack of ecological validity and the influence of the Hawthorne effect, which some observational studies may manage better.

Well-designed quasi-experiments and observational studies can still produce strong conclusions, but cannot be used by themselves to establish cause-and-effect conclusions.

The three main study designs

FIGURE 3.7: The three main study designs

3.6 External and internal validity

As far as possible, all studies should be designed to be externally valid (Chap. 5) and internally valid (Chaps. 7 and 8)

Internally validity refers to how reasonable and logical it is to conclude that changes in the value of the response variable can be attributed to changes in the value of the explanatory variable; that is, it refers to the strength of the inferences made from the study.

Studies with high internal validity show that changes in the response variable can confidently be related to changes in the explanatory variable in the group that was studied; the possibility of other explanations has been minimised.

In contrast, studies with low internal validity leave open other possibilities, apart from the explanatory variable, to explain changes in the value of the response variable.

Definition 3.8 (Internal validity) Internally validity refers to how reasonable and logical it is to conclude that changes in the value of the response variable can be attributed to changes in the values of the explanatory variable; that is, the strength of the inferences made from the study.

A study with high internal validity shows that the changes in the response variable can be attributed to changes in the explanatory variables; other explanations have been ruled out.

Example 3.10 (Low internal validity) A study of programs that used double-fortified salt programs to manage iodine and iron deficiencies examined numerous existing studies. The authors found that

Internal validity of the efficacy trials was generally weak, with only 6 out of 22 studies rated as moderate or high quality. Primarily, RCTs [i.e., randomised controlled trials] included in this review had low internal validity because of issues around selection bias, unaccounted confounders, and participant withdrawals.

--- Leila M. Larson et al.102, p. 26S (emphasis added)

One of many threats to internal validity might be that the groups being compared are different to begin with (for example, if the group receiving echinacea is younger (on average) than the group receiving no medication). This is a form of confounding.

To check this, the baseline characteristics of the individuals in the groups can be compared: the groups being compared should be as similar as possible, so that any differences in the outcome cannot be attributed to pre-existing difference in the two groups being compared.

Example 3.11 (Baseline characteristics) In a study of treating depression in adults,103 three treatments were compared: exercise, basic body awareness therapy, or advice.

If any differences between the treatments were found, the researchers need to be confident that the differences were due to the treatment.

For this reason, the three groups were compared to ensure the groups were similar in terms of average ages, percentage of women, taking of anti-depressants, and many other aspects.

An internally valid study requires studies to be carefully designed; this is discussed at length later (Chaps. 7 and 8). In general, well-designed experimental studies are more likely to be internally valid than observational studies (Fig. 3.8).

Well-designed true experiments are more likely to have high internal validity

FIGURE 3.8: Well-designed true experiments are more likely to have high internal validity

A study is externally valid if the results of the study are likely to generalised to other groups in the population, apart from those studied in the sample.

For a study to be externally valid it first needs to be internally valid, since the results must at least be internally valid for the group under study before being extended to other members of the population.

Using a random sample helps ensure external validity. In addition, the use of inclusion and exclusion criteria (Sect. 2.3.1) helps clarify to whom or what the results may apply outside of the sample being studied.

Definition 3.9 (External validity) Externally validity refers to the ability to generalise the results to other groups in the population, apart from the sample studied.

For a study to be truly externally valid, the sample must be a random sample from the population.

A study is externally valid if the results from the sample studied are likely to apply to the intended population. It does not mean that the results apply more widely than the intended population.

Example 3.12 (External validity) Suppose the population in a study is Queensland university students. The sample would be the students studied. The study is externally valid if the sample is a random sample from the population of Queensland university students.

The results will not necessarily apply to all Queensland residents, or university students outside of Queensland. However, this has nothing to do with externally validity. External validity concerns how the sample represents the intended population in the RQ, which is Queensland university students. The study is not concerned with all Queensland residents, or with non-Queensland university students.

3.7 The importance of design

Choosing the type of study is only a small part of research design. Planning the data collection process, and actually collecting the data, is still required. Data may be obtained by:

  • Using data already available: This is called secondary data.
  • Collecting new data: This is called primary data.

Either way, knowing how the data are obtained is important. The design phase is concerned with planning the best way to obtaining the data to ensure the study is internally and externally valid, as far as possible.

Internal validity considerations include:

  • What else might influence the values of the response variable, apart from the explanatory variable? (Chap. 6)
  • How can the study be designed effectively to maximise internal validity? (Chaps. 7 and 8)
  • How exactly will the data be collected? (Chap. 10)

External validity considerations include:

  • Sampling: Since we can't study the whole population, who or what do we study in the population (Chap. 5)? And how many do we need to study? (We need to learn more before we can answer this critical question in Chap. 5.)

Ethical issues must also be considered (Chap. 4), and the limitations of the study understood when the results are interpreted (Chap. 9).

The following short (humourous) video demonstrates the importance of understanding the design!

3.8 Summary

Studies may be:

  • descriptive (for descriptive RQs);
  • observational (for relational RQs); or
  • experimental (for interventional RQs).

Observational studies can usually be classified as retrospective, prospective, or cross-sectional. Experimental studies can usually be classified as true experiments or quasi-experiments.

Cause-and-effect conclusions can only be made from well-designed true experiments.

Ideally studies should be designed to be internally and externally valid. In general, experimental studies have better internal validity than observational studies.

Chapter 3 summary

FIGURE 3.9: Chapter 3 summary

The following short videos may help explain some of these concepts:

3.9 Quick review questions

  1. A study104 examined the 'red-light running behaviour of cyclists in Italy'. This study is most likely to be:

  2. A study of a sample whose results apply to the wider population of interest would be called:

  3. In a quasi-experiment, the researchers allocate treatments to groups that they have not organised. True or false?

  4. What is the difference between an true experiment and a quasi-experiment?

  5. A research study compared the use of two different education programs to reduce the percentage of patients experiencing ventilator-associated pneumonia (VAP).
    Paramedics from two cities were chosen to participate. Paramedics in City A were chosen (at random) to receive Program 1, and paramedics in the other city to receive Program 2.
    What type of study is this?

  6. Which of the following are true?

    • True experiments have a higher internal validity than observational studies.
    • Internal validity refers to the strength of the inferences made from the study.
    • Externally validity refers to the ability to generalise the results to other groups apart from those studied.
    • Inclusion and exclusion criteria can be used to clarify the internal validity.
    • Observational studies have a higher external validity than experimental studies.

Progress:

  1. Observational, since it would seem that the researchers are observing and not instructing cyclists to run red light...
  2. Externally valid: This is basically the definition of external validity.
  3. TRUE.
  4. See here.
  5. See here.
  6. See here.

3.10 Exercises

Selected answers are available in Sect. D.3.

Exercise 3.1 In a study on the shear strength of recycled concrete beams,105 beams were divided into three groups. Different loads were then applied to each group, and the shear strength needed to fracture the beams was measured.

Is this a quasi-experiment or a true experiment? Answering these questions may help:

  1. Do researchers allocate treatments to the groups?
  2. Do researchers allocate the who or what to groups?

Exercise 3.2 A study had this aim:

To compare the effectiveness of alternating pressure air mattresses vs. overlays, to prevent pressure ulcers.

--- Francisco Manzano et al.106 , p. 2099.

Patients were provided with either alternating pressure air overlays (in 2001) or alternating pressure air mattresses (in 2006). The number of pressure ulcers were recorded.

This study experimental, because the researchers provided the mattresses. Is this a true experiment or quasi-experiment? Explain.

Exercise 3.3 Consider this initial RQ (based on Erika Friedmann and Sue Thomas107), that clearly requires a lot of refining:

Are people with pets healthier?

To answer this RQ:

  1. Describe a useful and practical definition for P, O and C.
  2. Describe an experimental study to answer the RQ.
  3. Describe an observational study to answer the RQ.

Exercise 3.4 Consider this journal extract:

We randomly assigned 811 overweight adults to one of four diets [...] The diets consisted of similar foods and met guidelines for cardiovascular health [...] The primary outcome was the change in body weight after 2 years in [...] comparisons of low fat versus high fat and average protein versus high protein and in the comparison of highest and lowest carbohydrate content.

--- Frank M. Sacks et al.108, p. 859

  1. Define POCI.
  2. Is this study observational or experimental? Why?
  3. Is this study a quasi-experiment or a true experiment? Why?
  4. What are the units of analysis?
  5. What are the units of observation?
  6. What is the response variable?
  7. What is the explanatory variable?