Discussion: Limitations (19)
The items from STROBE state that you should report:
- Discuss limitations of the study, taking into account sources of potential bias or imprecision.
- Discuss both direction and magnitude of any potential bias
Some key items to consider adding:
- Describe the main limitations of the data sources and assessment methods (e.g., laboratory or collection procedures) used and implications for the interpretation of the findings
- Discuss implications of misclassification bias, unmeasured/residual confounding, missing data, and , selection factors for treatment, and changing eligibility over time
- Discuss the implications of using data that were not created or collected to answer the specific research question(s)
Explanation
The identification and discussion of the limitations of a study are an essential part of scientific reporting. It is important not only to identify the sources of bias and confounding that could have affected results, but also to discuss the relative importance of different biases, including the likely direction and magnitude of any potential bias (see also box 3, item 9).
Authors should also discuss any imprecision of the results. Imprecision may arise in connection with several aspects of a study, including the study size (item 10) and the measurement of exposures, confounders and outcomes (item 8). The inability to precisely measure true values of an exposure tends to result in bias towards unity: the less precisely a risk factor is measured, the greater the bias. This effect has been described as ‘attenuation’, (Fuller & Hidiroglou, 1978; Spearman, 1904) or more recently as ‘regression dilution bias’. (MacMahon et al., 1990) However, when correlated risk factors are measured with different degrees of imprecision, the adjusted relative risk associated with them can be biased towards or away from unity. (Greenland, 1980; Phillips & Smith, 1991, 1992)
When discussing limitations, authors may compare the study being presented with other studies in the literature in terms of validity, generalizability and precision. In this approach, each study can be viewed as contribution to the literature, not as a stand-alone basis for inference and action. (Poole et al., 2003) Surprisingly, the discussion of important limitations of a study is sometimes omitted from published reports. A survey of authors who had published original research articles in The Lancet found that important weaknesses of the study were reported by the investigators in the survey questionnaires, but not in the published article. (Horton, 2002; Vandenbroucke et al., 2007)
Example
- “Since the prevalence of counseling increases with increasing levels of obesity, our estimates may overestimate the true prevalence. Telephone surveys also may overestimate the true prevalence of counseling. Although persons without telephones have similar levels of overweight as persons with telephones, persons without telephones tend to be less educated, a factor associated with lower levels of counseling in our study. Also, of concern is the potential bias caused by those who refused to participate as well as those who refused to respond to questions about weight. Furthermore, because data were collected cross-sectionally, we cannot infer that counseling preceded a patient’s attempt to lose weight.” (Galuska et al., 1999; Vandenbroucke et al., 2007)
Field-specific guidance
Anti-microbial stewardship programs (Tacconelli et al., 2016)
- Provide description of sources of selection bias, including infection control measures, audit and confounding
Infectious disease molecular epidemiology (Field et al., 2014)
- Consider alternative explanations for findings when transmission chains are being investigated, and report the consistency between molecular and epidemiological evidence
Neonatal infections (Fitchett et al., 2016)
- Discuss sources of recruitment bias, particularly regarding the period of time shortly after birth. State source of denominator data and discuss possible related
Seroepidemiologic studies for influenza (Horby et al., 2017)
- Discuss limitations and strengths of the study with reference to Table 1
References
Field, N., Cohen, T., Struelens, M. J., Palm, D., Cookson, B., Glynn, J. R., Gallo, V., Ramsay, M., Sonnenberg, P., MacCannell, D., Charlett, A., Egger, M., Green, J., Vineis, P., & Abubakar, I. (2014). Strengthening the Reporting of Molecular Epidemiology for Infectious Diseases (STROME-ID): An extension of the STROBE statement. The Lancet Infectious Diseases, 14(4), 341–352. https://doi.org/10.1016/S1473-3099(13)70324-4
Fitchett, E. J. A., Seale, A. C., Vergnano, S., Sharland, M., Heath, P. T., Saha, S. K., Agarwal, R., Ayede, A. I., Bhutta, Z. A., Black, R., Bojang, K., Campbell, H., Cousens, S., Darmstadt, G. L., Madhi, S. A., Meulen, A. S.-t., Modi, N., Patterson, J., Qazi, S., … Lawn, J. E. (2016). Strengthening the Reporting of Observational Studies in Epidemiology for Newborn Infection (STROBE-NI): An extension of the STROBE statement for neonatal infection research. The Lancet Infectious Diseases, 16(10), e202–e213. https://doi.org/10.1016/S1473-3099(16)30082-2
Fuller, W. A., & Hidiroglou, M. A. (1978). Regression Estimation after Correcting for Attenuation. Journal of the American Statistical Association, 73(361), 99–104. https://doi.org/10.1080/01621459.1978.10480011
Galuska, D. A., Will, J. C., Serdula, M. K., & Ford, E. S. (1999). Are Health Care Professionals Advising Obese Patients to Lose Weight? JAMA, 282(16), 1576–1578. https://doi.org/10.1001/jama.282.16.1576
Greenland, S. (1980). The effect of misclassification in the presence of covariates. American Journal of Epidemiology, 112(4), 564–569. https://doi.org/10.1093/oxfordjournals.aje.a113025
Horby, P. W., Laurie, K. L., Cowling, B. J., Engelhardt, O. G., Sturm-Ramirez, K., Sanchez, J. L., Katz, J. M., Uyeki, T. M., Wood, J., Van Kerkhove, M. D., & the CONSISE Steering Committee. (2017). CONSISE statement on the reporting of Seroepidemiologic Studies for influenza (ROSES-I statement): An extension of the STROBE statement. Influenza and Other Respiratory Viruses, 11(1), 2–14. https://doi.org/10.1111/irv.12411
Horton, R. (2002). The Hidden Research Paper. JAMA, 287(21), 2775–2778. https://doi.org/10.1001/jama.287.21.2775
MacMahon, S., Peto, R., Collins, R., Godwin, J., MacMahon, S., Cutler, J., Sorlie, P., Abbott, R., Collins, R., Neaton, J., Abbott, R., Dyer, A., & Stamler, J. (1990). Blood pressure, stroke, and coronary heart disease: Part 1, prolonged differences in blood pressure: Prospective observational studies corrected for the regression dilution bias. The Lancet, 335(8692), 765–774. https://doi.org/10.1016/0140-6736(90)90878-9
Phillips, A. N., & Smith, G. D. (1991). How independent are "independent" effects? Relative risk estimation when correlated exposures are measured imprecisely. Journal of Clinical Epidemiology, 44(11), 1223–1231. http://www.sciencedirect.com/science/article/pii/0895435691901553
Phillips, A. N., & Smith, G. D. (1992). Bias in relative odds estimation owing to imprecise measurement of correlated exposures. Statistics in Medicine, 11(7), 953–961. https://doi.org/10.1002/sim.4780110712
Poole, C., Peters, U., Il’yasova, D., & Arab, L. (2003). Commentary: This study failed? International Journal of Epidemiology, 32(4), 534–535. https://doi.org/10.1093/ije/dyg197
Spearman, C. (1904). The proof and measurement of association between two things. Am J Psychol, 15, 72–101. http://doi.org/10.1037/11491-005
Tacconelli, E., Cataldo, M. A., Paul, M., Leibovici, L., Kluytmans, J., Schröder, W., Foschi, F., Angelis, G. D., Waure, C. D., Cadeddu, C., Mutters, N. T., Gastmeier, P., & Cookson, B. (2016). STROBE-AMS: Recommendations to optimise reporting of epidemiological studies on antimicrobial resistance and informing improvement in antimicrobial stewardship. BMJ Open, 6(2), e010134. https://doi.org/10.1136/bmjopen-2015-010134
Vandenbroucke, J. P., Elm, E. von, Altman, D. G., Gotzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. Epidemiology, 18(6), 805–835. https://doi.org/10.1097/EDE.0b013e3181577511