Assessment of transparency indicators across the biomedical literature: How open is open?
Open accessPublished version of a preprint
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Indicators of transparency and reproducibility across publications and time. (A) Indicators of transparency and reproducibility across 349 research articles (2015–2018). Most publications included COI or Funding disclosures, but few mentioned Data, Code, or Protocol sharing. Similarly, most claimed Novelty but few mentioned a Replication component. (B) Indicators of transparency and reproducibility across time on the basis of manual assessment. These graphs merge data from this study on 349 research articles (2015–2018) with similar data from 2 previous studies on another 590 PubMed articles (2000–2018) [20,22]. Proportions are displayed as a 4-year centered moving average. The shaded region indicates the 95% CI. The most notable change is that of COI disclosures, the reporting of which increased from 12% in 2000 to 76% in 2018. The data underlying this figure can be found on OSF at http://www.doi.org/10.17605/OSF.IO/E58WS. CI, confidence interval; COI, Conflict of interest; OSF, Open Science Framework.](https://figures.cochat.ai/paper-figures/2565f397135a445ebf5da54f39576a14/assessment-of-transparency-indicators-across-the-biomedical-literature-how-open-figure-1.jpg)





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“failures to document research protocols and to share data, materials (digital and physical), and code are hindering efforts to increase replicability”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Serghiou et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and data and code sharing has been the least reported aspect in many studies [25–30, 41].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In other articles, the authors only stated that the data could be made available on reasonable request, which is not counted as “current” data sharing in previous studies [20, 35].”
From Do randomised clinical trials on dental caries adopt Open Science practices? (2024) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“Our findings align with previous biomedical and dental research, which reported the presence of more articles with statement of conflict of interest and funding than data and code sharing and protocol registration [27–30].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [27]
In the citing paper
“Traditional peer review often fails to address widely accepted, but suboptimal, research practices and guideline details.”
From Is the future of peer review automated? (2022) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4 · Cited by 33,115
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., et al. (2019) · Journal of Open Source Software · doi:10.21105/joss.01686 · Cited by 22,138
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4_9 · Cited by 12,516
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Both theoretical expectations [3] and empirical evidence [4–8] suggest that most published results may be either non-reproducible or inflated.”
BackgroundSelf-citation1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1×
“Research reproducibility [1] is a fundamental tenet of science, yet recent reports suggest that reproducibility of published findings should not be taken for granted [2].”
Background
Research works printed but not linked: 5
- PubMed Central Tagging Guidelines. In: PubMed Central (PMC) [Internet]. [cited 20 Apr 2020]. Available from: https://www.ncbi.nlm.nih.gov/pmc/pmcdoc/tagging-guidelines/article/style.html
- D Ecer, 2017, ScienceBeam—using computer vision to extract PDF data, In: Elife
- Website. 2020 [cited 2020 Oct 29]. Available from: US National Library of Medicine. Policies. PubMed Central https://www.ncbi.nlm.nih.gov/pmc/about/guidelines/
- FTP Service. [cited 2020 Oct 29]. Available from: https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/
- JD Wallach, 2018, Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017., PLoS Biol, 16, e2006930, 10.1371/journal.pbio.2006930 · DOI · Paper page
Cited by
Items 101–140 of 140
Influence of social media on the download of radiology articles
de la Gala, D. H., Alcalá, E. S., Domènech-Ximenos, B., & Villar, C. G. (2025) · Radiología (English Edition) · doi:10.1016/j.rxeng.2024.08.002 · Cited by 1
Russo, L., Lentini, N., Soru, L., Pastorino, R., Boccia, S., & Ioannidis, J. P. A. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112338 · Cited by 1
Self-citationBetter, H., Asbi, T., Chaushu, G., Better, K., & Haim, D. (2026) · Frontiers in Oral Health · doi:10.3389/froh.2026.1901870 · Cited by 1
“The text-mining approach builds on prior automated assessments of transparency indicators (20); the present study differs in operating on full-text PDF, applying broader marker definitions, and covering a bounded specialty corpus with formal hypothesis testing.”
Open Science Policies and Practices Among Medical Journals
Vinatier, C., Dewi, A. P. M., Freyermuth, G., Millour, M., Scheidecker, M., Arnault, F.-J., Acher, M., Nachev, V., Weissgerber, T. L., DeVito, N., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.32070 · Cited by 2
Dulitzki, C., Crane, S. M., Hardwicke, T. E., & Ioannidis, J. P. A. (2024) · Royal Society Open Science · doi:10.1098/rsos.240016 · Cited by 3
“However, most primary datasets are not published alongside the articles that draw on them, nor provided when requested from the original authors [3,4].”
BackgroundSelf-citationPrerequisite for reproducible science: a call to embrace code sharing
Gun, A., & Garg, T. (2024) · The Lancet Regional Health - Southeast Asia · doi:10.1016/j.lansea.2024.100472 · Cited by 1
Silva, F. M., Adegboye, A. R. A., Naude, C. E., Curioni, C. C., Gomes, F. D. S., Collins, G. S., Kac, G., de Beyer, J. A., Cook, J. A., Ismail, L. C., et al. (2025) · American Journal of Clinical Nutrition · doi:10.1016/j.ajcnut.2025.01.016 · Cited by 2
“For instance, the proportion of research articles reporting funding and con fl icts of interest increased from 25% and 0% in 2000 to 89% and 91% in 2020, respectively [33].”
The use of the phrase “data not shown” in dental research
Raittio, E. J., Sofi‐Mahmudi, A., & Shamsoddin, E. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272695 · Cited by 2
Ioannidis, J. (2025) · medRxiv · doi:10.1101/2025.03.02.25323201 · Cited by 1
Self-citationA comprehensive analysis of code and data availability in biomedical researchPreprint
Deshpande, D., Sarkar, A. P., Guo, R., & Mangul, S. (2021) · doi:10.31219/osf.io/xjd8r
Meeting report on the Ninth International Congress on Peer Review and Scientific Publication
Yoo, Y. (2023) · Science Editing · doi:10.6087/kcse.303
Open science interventions proposed or implemented to assess researcher impact: a scoping review
Ghannad, M., Armond, A. C. V., Ng, J. Y., Ayala, A. P., Khan, H., Grossman, M. R., Cormack, G. V., Pham, B., Leeflang, M., Bossuyt, P. M., et al. (2023) · F1000Research · doi:10.12688/f1000research.140556.1
Assessment of transparency indicators in Space MedicinePreprint
Bellomo, R. K., Zavalis, E. A., & Ioannidis, J. P. A. (2023) · medRxiv · doi:10.1101/2023.12.01.23299278
Self-citationSupporting open science at PLOS Biology
Cadwallader, L., & Pariente, N. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002516
Digital Technology Ecotone to Revolutionize Health SectorPreprint
Coccia, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4852168
Research transparency in articles published in orthodontic journals: is it clear?
Yii, M. C. H., Cobourne, M. T., Pandis, N., & Seehra, J. (2025) · European Journal of Orthodontics · doi:10.1093/ejo/cjaf028
SPIRIT 2025 and CONSORT 2025 Statements: Guidance Tools to Ensure Clinical Trial Transparency
Dal‐Ré, R., & Caplan, A. L. (2025) · Archivos de Bronconeumología · doi:10.1016/j.arbres.2025.05.013
Interessenkonflikt und Verpflichtung und Forschungsintegrität
Redman, B. K. (2025) · Springer eBooks · doi:10.1007/978-3-031-92469-9_6
Rigor in Experimental Design in Nonregulated Preclinical Research in Neuroscience Drug Discovery
Steele, S., Bespalov, A., & Dequeker, E. M. C. (2025) · Wiley eBooks · doi:10.1002/9783527844678.ch6
Masoudi, F. (2025) · Journal of Rafsanjan University of Medical Sciences · doi:10.61882/jrums.24.7.667
Henssler, J. F., Reis‐Pardal, J., Koppel, L., & Ioannidis, J. P. A. (2025) · medRxiv · doi:10.1101/2025.10.20.25338369
Self-citationMonitoring of Open Science practices: a survey of 10 major medical journalsPreprint
Vinatier, C., Dewi, A. P. M., Freyermuth, G., Millour, M., Scheidecker, M., Arnault, F.-J., Acher, M., Nachev, V., Weissgerber, T. L., DeVito, N., et al. (2025) · medRxiv · doi:10.1101/2025.11.26.25341061
Vinatier, C., Fahed, E., Chollet, Y., Caquelin, L., Jaillard, S., den Eynden, V. V., Kozula, M., & Naudet, F. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00167-x · Cited by 1
“[6] If the answer is yes, you’ll no doubt be pleased that the scientific community is equipping itself with tools to make research more transparent and, ideally, more reproducible.”
Hamilton, D. G., Page, M. J., Finch, S., Everitt, S., & Fidler, F. (2022) · Figshare · doi:10.6084/m9.figshare.21532680
Hansford, H. J., Cashin, A. G., Bagg, M. K., Wewege, M. A., Ferraro, M. C., Kianersi, S., Mayo-Wilson, E., Grant, S., Toomey, E., Skinner, I., et al. (2022) · Figshare · doi:10.6084/m9.figshare.20445409
Borges, D. D. S., Mello, A. T., Nascimento, G. M., de Lima, L. P., de Camargo, A. M., Fiates, G. M. R., & Trindade, E. B. S. D. M. (2026) · Royal Society Open Science · doi:10.1098/rsos.251476
Lee, D. Y. H., Parker, A. J., Norbury, C., & Shanks, D. R. (2026) · Royal Society Open Science · doi:10.1098/rsos.250381 · Cited by 2
Henssler, J. F., Reis‐Pardal, J., Koppel, L., & Ioannidis, J. P. A. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112215 · Cited by 1
Self-citationOvercoming barriers to computational reproducibility
Hornung, R., Németh, L., Zadorozhnyi, O., Ullmann, T., Kammer, M., Killick, R., Paciorek, C., Chiquet, J., Herrmann, M., Batinović, L., et al. (2026) · Royal Society Open Science · doi:10.1098/rsos.252489 · Cited by 2
Moving from disclosure to substance in AI transparency
Zhan, H., & Moore, J. H. (2026) · Patterns · doi:10.1016/j.patter.2026.101585
Ioannidis, J. P. A. (2026) · European Journal of Clinical Investigation · doi:10.1111/eci.70240 · Cited by 1
Self-citationVinatier, C., Freyermuth, G., Millour, M., Lyan, G. L. B., Buet, S., Lunel, P., Robillard, N., Naudet, F., & Acher, M. (2026) · Open Research Europe · doi:10.12688/openreseurope.24615.1
“Rtransparent uses rule-based text mining to detect data sharing statements and conflict of interest disclosures, 12 while ODDPub targets data and code sharing using keyword-based classifiers.”
rtransparency: Identifies Indicators of Transparency
Serghiou, S., & Sofi‐Mahmudi, A. (2026) · doi:10.32614/cran.package.rtransparency
Self-citationGlobal landscape and temporal trends of colorectal surgical trials
AlSamhori, J. F., AlSamhori, A. R. F., AlSamhori, A. F., Odisho, T., Ali, A. A., & Shahait, A. D. (2026) · Il Giornale di Chirurgia - Journal of the Italian Association of Hospital Surgeons · doi:10.1097/ia9.0000000000000090
Balagam, P. K. (2026) · Research Square · doi:10.21203/rs.3.rs-11157883/v1
Metadata
- Authors
- Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- SciTech Strategies (United States) (US)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Yale University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-03-01
- Volume
- 19
- Issue
- 3
- Pages
- e3001107
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC7951980
- OpenAlex
- W3097500482
- Cited by
- 139
- References
- 42
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 29 · 2023: 33 · 2022: 30 · 2021: 13
- FWCI
- 22.39
- Type
- journal article
- Language
- en
- Update notices
- new_version (2021-03-11)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (HHSN271201800033C)
- Laura and John Arnold Foundation
- Center of Excellence in Regulatory Science and Innovation (U01FD005938)
- School of Medicine, Stanford University
- Stanford Data Science Institute, Stanford University
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- research transparency, reproducibility, data sharing, code sharing, conflict of interest disclosure, funding disclosure, protocol registration, biomedical literature, PubMed Central, open access publications, temporal trends
- MeSH terms
- Scholarly Communication, Humans, Publications, Reproducibility of Results, Databases, Factual, Conflict of Interest, Disclosure, Information Dissemination, Biomedical Research, Open Access Publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.