28.8 About practical importance and statistical significance
Hypothesis tests assess statistical significance, which answers the question: ‘Is there evidence of a difference between the value of the statistic and the value of the assumed parameter?’ Even very small differences between the sample statistic and the population parameter can be statistically different if the sample size is large enough.
In contrast, practical importance asks the question:
Is the difference between the value of the statistic and the value of the assumed parameter of any practical importance?
‘Practical importance’ and ‘statistical significance’ are two separate (but both important) issues. Whether a results is of practical importance depends upon the context: what the data are being used for, by whom, and for what purpose.
Example 28.2 (Practical importance) In the body-temperature study, very strong evidence exists that the mean body temperature had changed (‘statistical significance’).
But the change was so small, that for most purposes it has no practical importance. (There may be other (e.g., medical) situations where it does have practical importance however.)Example 28.3 (Practical importance) A study of some herbal medicines (Maunder et al. 2020) for weight loss found:
Phaseolus vulgaris resulted in a statistically significant weight loss compared to placebo, although this was not considered clinically significant.
In other words, although the difference in weight loss between placebo and Phaseolus vulgaris was unikely to be explained by chance (\(P<0.001\), which is ‘statistical significant’), the difference was so small in size (a mean weight loss of just 1.61 kg) that it was unlikely to be of any use in practice (‘practical importance’).
In this context, a weight loss of at least 2.5 kg was considered to be of practical importance.