Chapter 6 Estimating unwanted effects with a control group
6.1 Learning objectives
By the end of this chapter, you will be able to:
Explain the importance of control groups
Understand the strengths and weaknesses of study designs using untreated or active controls
The idea of a control group has already come up in Chapter 2, Chapter 4 and Chapter 5, where it has been noted that this provides one way of estimating how far an improvement from baseline to outcome is really due to the intervention. The logic is straightforward: if you have two groups – group A who has the intervention, and group B who does everything that group A does except for the intervention, then it should be possible to isolate the specific effect of the intervention. In this chapter we will discuss the different kinds of control group that can be used. But first, it is important to address a question of ethics.
6.2 Is it ethical to include a control group?
The argument against a control group goes something like this: we think our intervention works, but we need to do a study to establish just how well it works. It would be unethical to withhold the intervention from one group of people because they would then be deprived of a potentially helpful treatment. This may seem particularly important if the client has serious problems - shouldn’t we do all in our power to help?
This argument has as many holes in it as a Swiss cheese, and here’s why. Do you already know if the treatment works? If the answer is Yes, then why are you doing a study? Presumably it is because you think and hope it works, but don’t actually have evidence that it does. So you don’t know and you need to find out. Freedman (1987) coined the term clinical equipoise to describe the situation when there is genuine uncertainty among practitioners about whether an intervention is better or worse than no intervention (or another intervention) - it is only ethical to do a trial of intervention when this is the case. But this can be hard to judge.
Unfortunately, the kinds of systematic bias discussed in Chapter 5 can lead practitioners to have an unwavering conviction that their intervention is effective: they see improvement in the clients they work with, and assume it is because of what they have done. The harsh reality is that when properly controlled studies are conducted, it is often found that the intervention is not effective and that the improvement is just as great in controls as in the intervention group. There are numerous examples of this in mainstream medicine, as well as in education and allied health disciplines.
6.3 Possible adverse effects of intervention
Worse still, some interventions do more harm than good. The need to evaluate adverse impacts of intervention is well-recognized in mainstream medicine, where it is easy to see the increased risks of morbidity or mortality, or unpleasant side effects. In clinical trials, a new treatment or medication will pass through typically five stages or phases of trials. A each stage the number of participants is increased, and the early trials are specifically designed to test safety and efficacy, beginning with very small doses. Typically for drug studies, before human trials are attempted, treatments undergo substantial lab-based evaluation and sometimes animal testing.
In fields such as education or speech and language therapy it is typically assumed that interventions won’t do any harm – and it is true that types of adverse effect seen in medicine are not likely. Nevertheless, any intervention creates opportunity costs – the time (and sometimes money) that are spent in intervention can’t be spent doing other things. In the study of computerized intervention by Bishop et al. (2006) head teachers were generally keen to take part, but they would ask what activities children should miss if they were to spend 15 minutes every day on our computerized tasks for a period of six weeks. Should we ask children to forfeit their breaks, or to skip non-academic activities such as swimming? This would hardly be popular with the children, we felt. Or should they take time out of lessons in English or Maths?
The Arrowsmith ProgramTM
The Arrowsmith Program is marketed as a neuroscience-based intervention for children with a wide range of specific learning disabilities. A review of the research literature on Arrowsmith finds no evidence of effectiveness, despite the program having been in existence since the 1980s (Bowen, 2020). One of the most comprehensive reviews of Arrowsmith research is in the D.Phil. thesis of Debra Kemp-Koo from the University of Saskatchewan in 2013. In her introduction, Dr Kemp-Koo included an account of a study of children attending the private Arrowsmith school in Toronto:
- All of the students in the study completed at least one year in the Arrowsmith program with most of them completing two years and some of them completing three years. At the end of the study many students had completed their Arrowsmith studies and left for other educational pursuits. The other students had not completed their Arrowsmith studies and continued at the Arrowsmith School. Most of the students who participated in the study were taking 6 forty minute modules of Arrowsmith programming a day with 1 forty minute period a day each of English and math at the Arrowsmith School. Some of the students took only Arrowsmith programming or took four modules of Arrowsmith programming with the other half of their day spent at the Arrowsmith school or another school in academic instruction (p. 34-35; our emphasis).
So children at Arrowsmith schools were spending one to three years working on unevidenced, repetitive exercises, rather than the regular academic curriculum. As Kemp-Koo (2013) remarked:
- The Arrowsmith program itself does not focus on academic instruction, although some of these students did receive some academic instruction apart from their Arrowsmith programming. The length of time away from academic instruction could increase the amount of time needed to catch up with the academic instruction these students have missed. (p. 35; our emphasis).
It seems remarkable that children who have a range of specific learning difficulties are described as needing catch-up academic instruction because they have missed regular classroom activities because of time spent on the Arrowsmith program. This is a classic case of an intervention that carries a substantial opportunity cost - quite apart from the financial cost to families.
When we move to consider interventions outside the school setting, there are still time costs involved: an aphasic adult with poor mobility or a harassed mother with three small children who is reliant on public transport may have considerable difficulty attending clinic appointments. Of course, even if the intervention is ineffective, the clients may be glad to have the attention and professional interest of a therapist. But if that is something that is of importance to them, then that needs to be included in the outcome measures – i.e. it is necessary to demonstrate it is so, rather than just assuming this is the case.
For parent training programmes, there is an additional risk of making parents worry that they are inadequate, that they don’t have time to devote to activities with their child, or that their child’s difficulties are all their fault. Clearly, competent professionals will be at pains to counteract that perception, but it is an easy mindset to fall into (Weiss, 1991).
Finally, the person who is the focus of intervention may feel embarrassed, stressed or stigmatized by being singled out for special treatment – this is most likely to be a potential problem for children who are pulled out of regular classes for intervention sessions. There is a poignant account of such an experience by the dyslexic chef Jamie Oliver who described how other children would sing “Special Needs, Special Needs” to the tune of Let it Be when he and another child were pulled out of class for individual tuition. Here again, a good professional will recognize that risk and do their best to address the concerns, but this needs to be thought about, rather than blithely assuming that because the intention of the therapist is good and helpful, their impact has to be beneficial.
All these potentially adverse impacts, including large investments of time, will generally be seen as a price well worth paying if the intervention is effective. At the end of the day, a decision whether to intervene or not should involve an appraisal of costs and benefits. Problems arise when costs are ignored, and a rosy, but unevidenced, view of benefits is assumed.
6.4 Ethics of uncontrolled studies
As explained in Chapter 5, a study that just compares people on a language measure before and after intervention is generally uninterpretable, because there are numerous factors that could be responsible for change. Having a control group is not the only way forward: in Chapter 20, we discuss other types of research design that may be more appropriate for specific contexts. But in most situations where the aim is to test the effectiveness of an intervention for a particular condition, a study with a control group will be the best way to get useful information. Indeed, doing a study without controls is unethical if you end up investing the time of clients and professionals, and research funds, doing a study that cannot answer the question of whether the intervention is effective.
6.5 Treated vs untreated controls
As noted above, a common design is to compare group A, who receive intervention, with group B who are treated just the same but without any intervention. If you see no effect of intervention using this design, and the study is big enough to give adequate statistical power (see Chapter 13), then you can be pretty certain that the intervention is not effective. But if you do see a reliable improvement in group A, over and above and change in group B, can you be certain it was the intervention that made the difference?
There are two things to be concerned about here. First, in the previous chapter we discussed the kinds of very general impact that can be caused just by being in an intervention group. These work in the opposite way to the possible adverse effects discussed above: although some people may be embarrassed or stressed by being a target of intervention, others may be reassured or made more confident. A good therapist will do more than just administer an intervention in a standard fashion: they will form a relationship with their client which can itself be therapeutic. This, however, creates a confound if the control group, B, just has “business as usual” because it means we are really comparing group A, who gets a specific intervention plus a therapeutic relationship, with group B, who gets nothing.
A creative way of dealing with this confound is to ensure that group B does also get some kind of intervention, focusing on something that is worthwhile, but different from the one that is the focus of the study. This is known as an active control. It is important to choose a comparison intervention that is not expected to have an impact on the main outcome measure – otherwise you could find yourself in a situation where everyone gets better but groups A and B don’t differ, and you are left uncertain whether both interventions worked or neither did. On the other hand, it would be unethical to use an active control that was of no conceivable benefit. An example from the speech and language field is the study by Burgoyne et al. (2018). They compared preschoolers who had a parent-based language intervention with a group who had a parent-based intervention designed to improve motor skills. Another example comes from McGillion et al. (2017), who compared an intervention designed to increase caregiver contingent talk to infants with a control condition where caregivers viewed a video about healthy eating and tooth-brushing and were asked to spend 15 min a day with their child making these a habit.
The use of a contrasting intervention overcomes another possible concern about untreated controls, which is those in group B may actually do worse because they know they are missing out on an intervention that might be able to help them (see Chapter 4).
Another popular approach to controls is to have a two-stage study. Stage 1 is a classic controlled study where a treated group A is compared with an untreated group B. After the intervention phase, group B is then offered the intervention. At this point, group A may continue with the intervention, or intervention may be withdrawn. These kind of designs are called crossover or wait-list designs and will be discussed further in Chapter 19.
6.6 Check your understanding
Take one of the three vignettes from Chapter 1, or use an intervention method you are familiar with.
- Make a list of any possible harms you can think of that might be caused by the intervention. Remember that in behavioural areas, most harms are likely to be psychological, social or financial.
- If you wanted to evaluate this intervention using an active control group (rather than an untreated control), what kind of intervention could you use for the controls?