Discussing study limitations in reports of biomedical studies- the need for more transparency
Open accessAbstract
Unbiased and frank discussion of study limitations by authors represents a crucial part of the scientific discourse and progress. In today's culture of publishing many authors or scientific teams probably balance 'utter honesty' when discussing limitations of their research with the risk of being unable to publish their work. Currently, too few papers in the medical literature frankly discuss how limitations could have affected the study findings and interpretations. The goals of this commentary are to review how limitations are currently acknowledged in the medical literature, to discuss the implications of limitations in biomedical studies, and to make suggestions as to how to openly discuss limitations for scientists submitting their papers to journals. This commentary was developed through discussion and logical arguments by the authors who are doing research in the area of hedging (use of language to express uncertainty) and who have extensive experience as authors and editors of biomedical papers. We strongly encourage authors to report on all potentially important limitations that may have affected the quality and interpretation of the evidence being presented. This will not only benefit science but also offers incentives for authors: If not all important limitations are acknowledged readers and reviewers of scientific articles may perceive that the authors were unaware of them. Authors should take advantage of their content knowledge and familiarity with the study to prevent misinterpretations of the limitations by reviewers and readers. Articles discussing limitations help shape the future research agenda and are likely to be cited because they have informed the design and conduct of future studies. Instead of perceiving acknowledgment of limitations negatively, authors, reviewers and editors should recognize the potential of a frank and unbiased discussion of study limitations that should not jeopardize acceptance of manuscripts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Limitations are unavoidable in scientific research, and describing them is essential to share experience, guide best practice, and aid the design of future experiments.
“Limitations are unavoidable in scientific research, and describing them is essential to share experience, guide best practice, and aid the design of future experiments [228].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [228]
The Introduction (page not provided) calls candid discussion of limitations crucial to scientific discourse and says it informs higher-quality future studies. The Discussion section (page not provided) recommends using limitations to guide future research and study design. The copy does not establish that limitations are unavoidable in all scientific research or specifically say that describing them guides best practice. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305 · see citing works
What is ‘quality of evidence’ and why is it important to clinicians?
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39490.551019.be · Cited by 3,250 · see citing works
Users' guide to detecting misleading claims in clinical research reports
Montori, V. M., Jaeschke, R., Schünemann, H. J., Bhandari, M., Brozek, J. L., Devereaux, P. J., & Guyatt, G. H. (2004) · BMJ · doi:10.1136/bmj.329.7474.1093 · Cited by 106 · cited 1× · see citing works
“”[1] We think that, in today’s culture of publishing biomedical studies, many authors may not want to discuss limitations of their studies because they perceive a transparency threshold beyond which the probability of manuscript acceptance goes down (perhaps even to zero) [2].”
Background
Flagged first, then by citations
Research works printed but not linked: 9
- Ioannidis JP Limitations are not properly acknowledged in the scientific literature J Clin Epidemiol 2007 60 4 324 329 10.1016/j.jclinepi.2006.09.011 17346604 · DOI · Paper page
- Puhan MA Heller N Joleska I Siebeling L Muggensturm P Umbehr M Goodman S ter Riet G Acknowledging Limitations in Biomedical Studies: The ALIBI Study The Sixth International Congress on Peer Review and Biomedical Publication 2009 Vancouver, Canada: JAMA and BMJ
- Rothman KJ Greenland S Lash TL Modern Epidemiology 2008 3 Lippincott Williams & Wilkins
- Clark S Horton R Putting research into context--revisited Lancet 2010 376 9734 10 11 10.1016/S0140-6736(10)61001-X 20609979 · DOI
- Puhan MA Guyatt GH Goldstein R Mador J McKim D Stahl E Griffith L Schunemann HJ Relative responsiveness of the Chronic Respiratory Questionnaire, St. Georges Respiratory Questionnaire and four other health-related quality of life instruments for patients with chronic lung disease Respir Med 2007 101 2 308 316 10.1016/j.rmed.2006.04.023 16782320 · DOI
- Teckle P Peacock S McTaggart-Cowan H van der Hoek K Chia S Melosky B Gelmon K The ability of cancer-specific and generic preference-based instruments to discriminate across clinical and self-reported measures of cancer severities Health Qual Life Outcomes 2011 9 106 10.1186/1477-7525-9-106 22123196 PMC3236471 · DOI
- Haynes RB Sackett D Tugwell P Guyatt GH Clinical Epidemiology: How to Do Clinical Practice Research 2005 3 Lippincott Williams & Wilkins
- Bilotta C Bowling A Nicolini P Case A Pina G Rossi SV Vergani C Older People's Quality of Life (OPQOL) scores and adverse health outcomes at a one-year follow-up. A prospective cohort study on older outpatients living in the community in Italy Health Qual Life Outcomes 2011 9 72 10.1186/1477-7525-9-72 21892954 PMC3175435 · DOI
- Bausewein C Simon ST Benalia H Downing J Mwangi-Powell FN Daveson BA Harding R Higginson IJ Implementing patient reported outcome measures (PROMs) in palliative care--users' cry for help Health Qual Life Outcomes 2011 9 27 10.1186/1477-7525-9-27 21507232 PMC3112059 · DOI
Cited by
Sorted by citations
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677 · see citing works
- Authors
- Puhan, M. A., Akl, E. A., Bryant, D., Xie, F., Apolone, G., & Riet, G.
- Affiliations
- Johns Hopkins University (US)
- University at Buffalo, State University of New York (US)
- Western University (CA)
- McMaster University (CA)
- Amsterdam UMC Location University of Amsterdam (NL)
- Corresponding author
- Puhan, Milo A
- Venue
- Health and Quality of Life Outcomes
- Published
- 2012
- Volume
- 10
- Issue
- 1
- Pages
- 23
- ISSN
- 1477-7525 (ISSN-L)
- PMC
- PMC3305390
- OpenAlex
- W2099563950
- Cited by
- 54
- References
- 3
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 9 · 2023: 6 · 2022: 5 · 2021: 3 · 2020: 7 · 2019: 4 · 2017: 1 · 2015: 1 · 2014: 4 · 2013: 1 · 2012: 1
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Health and Quality of Life Outcomes (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- research limitations, reporting transparency, hedging, publication bias, peer review, manuscript acceptance, research agenda setting
- MeSH terms
- Humans, Publishing, Quality Control, United States, Needs Assessment, Biomedical Research, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
