2.4 Limitations: Ecological validity

The practicality of the study results in the real world should also be discussed. This is called ecological validity.

Definition 2.1 (Ecological validity) A study is ecologically valid if the study methods, materials and context approximate the real situation being studied.

Studies don't need to be ecologically valid to be useful; much can be learnt under special conditions, as long as the potential limitations are understood when applying the results to the real world. Although ecological validity is not essential for a good study, ecological validity is useful if it is possible to achieve.

The ecological validity of experimental studies may be compromised because the experimental conditions are sometimes contrived.

Example 2.5 (Ecological validity) Consider a study to determine how likely it is that people will buy a coffee in a reuseable cup.

We could ask people about their intentions. This study may not be ecologically valid, as how people act in the real world may not align with what the say, especially when social pressures exist to use reusable cups.

An alternative study involves watching people buy coffees at various coffee shops, and record what people actually do in practice.

This second study is more likely to be ecologically valid, as we are watching actual behaviour in the real world.

A study was completed to observe the effect of using high-mounted rear brake lights (Kahane and Hertz 1998), which are now commonplace. The American study showed that such lights reduced rear-end collisions by about 50%. However, after making these lights mandatory, rear-end collisions reduced by only 5%. Why?

References

Kahane, Charles J., and Ellen Hertz. 1998. “The Long-Term Effectiveness of Center High Mounted Stop Lamps in Passenger Cars and Light Trucks.” DOT HT 8087 696. NHTSA. https://one.nhtsa.gov/cars/rules/regrev/evaluate/808696.html.