Reproducible Research Practices and Transparency across the Biomedical Literature
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
There is a growing movement to encourage reproducibility and transparency practices in the scientific community, including public access to raw data and protocols, the conduct of replication studies, systematic integration of evidence in systematic reviews, and the documentation of funding and potential conflicts of interest. In this survey, we assessed the current status of reproducibility and transparency addressing these indicators in a random sample of 441 biomedical journal articles published in 2000-2014. Only one study provided a full protocol and none made all raw data directly available. Replication studies were rare (n = 4), and only 16 studies had their data included in a subsequent systematic review or meta-analysis. The majority of studies did not mention anything about funding or conflicts of interest. The percentage of articles with no statement of conflict decreased substantially between 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles reporting statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0% in 2014) increased. Articles published in journals in the clinical medicine category versus other fields were almost twice as likely to not include any information on funding and to have private funding. This study provides baseline data to compare future progress in improving these indicators in the scientific literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the absence of stringent community norms or regulations, scientific claims are regularly published without public release of the research data upon which they are based [7,9,14].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the literature contains studies with questionable methodologies, missing or incomplete data, false or exaggerated effect measures, and many studies do not mention funding or potential conflicts of interest [4–8].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, assessment of analytic reproducibility is typically precluded by poor access to research data [7–14].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Replication validity of genetic association studies
Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., & Contopoulos-Ioannidis, D. G. (2001) · Nature Genetics · doi:10.1038/ng749 · Cited by 1,832
Self-citationScope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationAnalysis of DNA methylation in cancer: location revisited
Koch, A., Joosten, S. C., Feng, Z., de Ruijter, T. C., Draht, M. X., Melotte, V., Smits, K. M., Veeck, J., Herman, J. G., Van Neste, L., et al. (2018) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-018-0004-4 · Cited by 710
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700
“The result is that a tiny minority of data from published articles are shared [7,20].”
BackgroundA solution to minimum sample size for regressions
Jenkins, D. G., & Quintana-Ascencio, P. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229345 · Cited by 665
“Small N has been discussed as one part of a larger reproducibility problem, where recommended solutions include registered studies, conflict of interest statements, and publication of data [11,13,14,15,16,17].”
Background
Metadata
- Authors
- Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A.
- Affiliations
- Emory University (US)
- Stanford Medicine (US)
- Centers for Disease Control and Prevention (US)
- National Institutes of Health (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-01-04
- Volume
- 14
- Issue
- 1
- Pages
- e1002333
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4699702
- OpenAlex
- W2228927609
- Cited by
- 310
- References
- 41
- Citations by year
- 2026: 12 · 2025: 16 · 2024: 19 · 2023: 26 · 2022: 19 · 2021: 38 · 2020: 33 · 2019: 40 · 2018: 42 · 2017: 31 · 2016: 33
- FWCI
- 92.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- reproducibility, research transparency, biomedical literature, data availability, raw data sharing, replication study, systematic reviews, funding disclosure, conflicts of interest
- MeSH terms
- Periodicals as Topic, Reproducibility of Results, Conflict of Interest, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

