All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Acknowledgment of all serious limitations to research evidence is important for patient care and scientific progress. Formal research on how biomedical authors acknowledge limitations is scarce. OBJECTIVES: To assess the extent to which limitations are acknowledged in biomedical publications explicitly, and implicitly by investigating the use of phrases that express uncertainty, so-called hedges; to assess the association between industry support and the extent of hedging. DESIGN: We analyzed reporting of limitations and use of hedges in 300 biomedical publications published in 30 high and medium -ranked journals in 2007. Hedges were assessed using linguistic software that assigned weights between 1 and 5 to each expression of uncertainty. RESULTS: Twenty-seven percent of publications (81/300) did not mention any limitations, while 73% acknowledged a median of 3 (range 1-8) limitations. Five percent mentioned a limitation in the abstract. After controlling for confounders, publications on industry-supported studies used significantly fewer hedges than publications not so supported (p = 0.028). LIMITATIONS: Detection and classification of limitations was--to some extent--subjective. The weighting scheme used by the hedging detection software has subjective elements. CONCLUSIONS: Reporting of limitations in biomedical publications is probably very incomplete. Transparent reporting of limitations may protect clinicians and guideline committees against overly confident beliefs and decisions and support scientific progress through better design, conduct or analysis of new studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698
CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000251 · Cited by 2,296
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
Self-citationYank, V., Rennie, D., & Bero, L. A. (2007) · BMJ · doi:10.1136/bmj.39376.447211.be · Cited by 235 · cited 2×
“Fletcher and Black discussing several ways in which ‘‘spin’’ occurs in the medical literature suggested that editors be more vigilant with industry-sponsored studies [9], while Yank et al, studying meta-analyses, found that for-profit funding was associated with more favorable conclusions, which were not accounted for…”
BackgroundLimitations are not properly acknowledged in the scientific literature
Ioannidis, J. P. A. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.09.011 · Cited by 180 · cited 3×
“However, he used an automated search strategy on the texts that is likely to have missed acknowledgments of limitations [3].”
BackgroundMethodInfluentialHorton, R. (2002) · JAMA · doi:10.1001/jama.287.21.2775 · Cited by 75
Discussing study limitations in reports of biomedical studies- the need for more transparency
Puhan, M. A., Akl, E. A., Bryant, D., Xie, F., Apolone, G., & Riet, G. (2012) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-10-23 · Cited by 54
Self-citation
Cited by
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
Is the future of peer review automated?
Schulz, R., Barnett, A., Bernard, R., Brown, N. J. L., Byrne, J. A., Eckmann, P., Gazda, M. A., Kilicoglu, H., Prager, E. M., Salholz-Hillel, M., et al. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06080-6 · Cited by 80
“Examples include incomplete reporting of criteria to assess risk of bias [6], ambiguous or incorrect citations [29], lack of open data or code [30], incorrect statistical calculations [31], and underreporting of ethics [32], sex as a biological variable [33], and limitations statements [34].”
BackgroundBiomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Metadata
- Authors
- ter Riet, G., Chesley, P., Gross, A. G., Siebeling, L., Muggensturm, P., Heller, N., Umbehr, M., Vollenweider, D., Yu, T., Akl, E. A., Brewster, L., Dekkers, O. M., Mühlhauser, I., Richter, B., Singh, S., Goodman, S., & Puhan, M. A.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-11-20
- Volume
- 8
- Issue
- 11
- Pages
- e73623
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2022527135
- Cited by
- 37
- References
- 8
- FWCI
- 2.15
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies. https://app.cochat.ai/papers/10.1371/journal.pone.0073623
- The paper
- ter Riet, G., Chesley, P., Gross, A. G., Siebeling, L., Muggensturm, P., Heller, N., Umbehr, M., Vollenweider, D., Yu, T., Akl, E. A., Brewster, L., Dekkers, O. M., Mühlhauser, I., Richter, B., Singh, S., Goodman, S., & Puhan, M. A. (2013). All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies. PLoS ONE. https://doi.org/10.1371/journal.pone.0073623
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.