4.10 Validity Evidence and Inferences

So far, we have explored two key aspects of a study’s design. Sampling referes to how participants are selected from a larger population (like sampling a soup). Assignment refers to how the selected participants are assigned to comparison groups. Both sampling and assignment play a large role in the inferences that can be drawn. Please read the link below for a refresher:

https://www.statology.org/random-selection-vs-random-assignment/

Validity is the degree to which inferences and conclusions are meaningful and accurate. It is important that you note that the validation is about the inferences, not the study. (You should never claim a study is valid.) Unfortunately, it is almost impossible to know whether the inferences we draw are valid or not. Therefore, the best we can do is to provide evidence that supports the claims of validity and present that evidence to people. In research studies there are two major types of validity evidence that should be considered:

Internal Validity Evidence: This is evidence that supports the drawing of cause-and-effect conclusions. To assess the internal validity evidence, we consider three criteria:

  1. Association: Is the cause associated with the effect
  2. Timing: Did the cause come before the effect?
  3. Plausible alternative explanations: Can we rule out plausible alternative explanations?

Read more at:

External Validity Evidence: This is evidence that supports generalizing conclusions to a broader population beyond the sample that was studies. To assess the external validity evidence, we consider two criteria:

  1. Representativeness: Is the sample represenatitve of the population?
  2. Uncertainty: Have the researchers accounted for uncertainty in the sample?

Read more at:

Much like evidence used in a court of law, it is useful to think about validity evidence on a continuum. For example, after reading about the sampling and design of a particular study you might envision the following two “barometers” of the internal and external validity evidence for a study’s inferences:



For the study depicted above, the inferences may generalize, but are probably not causal. As you read research studies (and work on the class activities) you will learn how to situate the yellow dot along these two continuums by considering the sampling and design plans used in the study.

Read more about random sampling, random assignment, and validity evidence at https://www.statology.org/random-selection-vs-random-assignment/