Assessment of transparency indicators across the biomedical literature: How open is open?
Open accessPublished version of a preprint
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Indicators of transparency and reproducibility across publications and time. (A) Indicators of transparency and reproducibility across 349 research articles (2015–2018). Most publications included COI or Funding disclosures, but few mentioned Data, Code, or Protocol sharing. Similarly, most claimed Novelty but few mentioned a Replication component. (B) Indicators of transparency and reproducibility across time on the basis of manual assessment. These graphs merge data from this study on 349 research articles (2015–2018) with similar data from 2 previous studies on another 590 PubMed articles (2000–2018) [20,22]. Proportions are displayed as a 4-year centered moving average. The shaded region indicates the 95% CI. The most notable change is that of COI disclosures, the reporting of which increased from 12% in 2000 to 76% in 2018. The data underlying this figure can be found on OSF at http://www.doi.org/10.17605/OSF.IO/E58WS. CI, confidence interval; COI, Conflict of interest; OSF, Open Science Framework.](https://figures.cochat.ai/paper-figures/2565f397135a445ebf5da54f39576a14/assessment-of-transparency-indicators-across-the-biomedical-literature-how-open-figure-1.jpg)





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“failures to document research protocols and to share data, materials (digital and physical), and code are hindering efforts to increase replicability”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Serghiou et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and data and code sharing has been the least reported aspect in many studies [25–30, 41].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In other articles, the authors only stated that the data could be made available on reasonable request, which is not counted as “current” data sharing in previous studies [20, 35].”
From Do randomised clinical trials on dental caries adopt Open Science practices? (2024) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“Our findings align with previous biomedical and dental research, which reported the presence of more articles with statement of conflict of interest and funding than data and code sharing and protocol registration [27–30].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [27]
In the citing paper
“Traditional peer review often fails to address widely accepted, but suboptimal, research practices and guideline details.”
From Is the future of peer review automated? (2022) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4 · Cited by 33,115
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., et al. (2019) · Journal of Open Source Software · doi:10.21105/joss.01686 · Cited by 22,138
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Wickham, H. (2016) · Use R! · doi:10.1007/978-3-319-24277-4_9 · Cited by 12,516
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Both theoretical expectations [3] and empirical evidence [4–8] suggest that most published results may be either non-reproducible or inflated.”
BackgroundSelf-citation1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1×
“Research reproducibility [1] is a fundamental tenet of science, yet recent reports suggest that reproducibility of published findings should not be taken for granted [2].”
Background
Research works printed but not linked: 5
- PubMed Central Tagging Guidelines. In: PubMed Central (PMC) [Internet]. [cited 20 Apr 2020]. Available from: https://www.ncbi.nlm.nih.gov/pmc/pmcdoc/tagging-guidelines/article/style.html
- D Ecer, 2017, ScienceBeam—using computer vision to extract PDF data, In: Elife
- Website. 2020 [cited 2020 Oct 29]. Available from: US National Library of Medicine. Policies. PubMed Central https://www.ncbi.nlm.nih.gov/pmc/about/guidelines/
- FTP Service. [cited 2020 Oct 29]. Available from: https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/
- JD Wallach, 2018, Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017., PLoS Biol, 16, e2006930, 10.1371/journal.pbio.2006930 · DOI · Paper page
Cited by
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
“, 2021b), failures to document research protocols and to share data, materials (digital and physical), and code are hindering efforts to increase replicability (see also Lemmon et al., 2014; Serghiou et al., 2021; Stodden et al., 2018; Vines et al., 2014).”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
Is the future of peer review automated?
Schulz, R., Barnett, A., Bernard, R., Brown, N. J. L., Byrne, J. A., Eckmann, P., Gazda, M. A., Kilicoglu, H., Prager, E. M., Salholz-Hillel, M., et al. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06080-6 · Cited by 80
“Examples include incomplete reporting of criteria to assess risk of bias [6], ambiguous or incorrect citations [29], lack of open data or code [30], incorrect statistical calculations [31], and underreporting of ethics [32], sex as a biological variable [33], and limitations statements [34].”
BackgroundHamilton, D. G., Hong, K., Fraser, H., Rowhani-Farid, A., Fidler, F., & Page, M. J. (2023) · BMJ · doi:10.1136/bmj-2023-075767 · Cited by 86
Open Science 2.0: Towards a truly collaborative research ecosystem
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023) · PLOS Biology · doi:10.1371/journal.pbio.3002362 · Cited by 72
Quantitative research assessment: using metrics against gamed metrics
Ioannidis, J. P. A., & Maniadis, Z. (2023) · Internal and Emergency Medicine · doi:10.1007/s11739-023-03447-w · Cited by 75
“For example, a free, publicly available resource covers all the open-access biomedical literature for these indicators [68].”
Self-citationWhite, N. M., Balasubramaniam, T., Nayak, R., & Barnett, A. G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0264360 · Cited by 76
Biosecurity in an age of open science
Smith, J. A., & Sandbrink, J. B. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001600 · Cited by 64
“Open science aims to increase the reliability and efficiency of scientific research [1,2].”
BackgroundPownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z. M., Grinschgl, S., Elsherif, M. M., Gilligan-Lee, K. A., Oliveira, C. M., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221255 · Cited by 78
Page, M. J., Nguyen, P., Hamilton, D. G., Haddaway, N., Kanukula, R., Moher, D., & McKenzie, J. E. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.03.003 · Cited by 59
Prognostic Biomarkers in Kidney Transplantation: A Systematic Review and Critical Appraisal
Raynaud, M., Awadhi, S. A., Louis, K., Zhang, H., Su, X., Goutaudier, V., Wang, J., Demir, Z., Wei, Y., Truchot, A., et al. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000260 · Cited by 50
“19 These indicators are code sharing, data sharing, protocol registration, and con fl icts of interest disclosures.”
BackgroundSelf-citationMedical journal requirements for clinical trial data sharing: Ripe for improvement
Naudet, F., Siebert, M., Pellen, C., Gaba, J., Axfors, C., Cristea, I., Danchev, V., Mansmann, U., Ohmann, C., Wallach, J. D., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003844 · Cited by 43
“Recent efforts have been made to monitor data sharing, and other open science practices, in journal articles [19], and these tools (https://github.”
BackgroundSelf-citationGuidance for the reporting of bibliometric analyses: A scoping review
Ng, J. Y., Liu, H., Masood, M., Syed, N., Stephen, D. M., Ayala, A. P., Sabé, M., Solmi, M., Waltman, L., Haustein, S., et al. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.12 · Cited by 60
An analysis of the effects of sharing research data, code, and preprints on citations
Colavizza, G., Cadwallader, L., LaFlamme, M., Dozot, G., Lecorney, S., Rappo, D., & Hrynaszkiewicz, I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0311493 · Cited by 61
“While the uptake of these practices by researchers varies by field, career stage, and region, their prevalence is growing overall [7, 8].”
Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Tzavella, L., Bendixen, T., Handcock, S. A., Köneke, V. E., & Ioannidis, J. P. A. (2022) · Royal Society Open Science · doi:10.1098/rsos.220139 · Cited by 45
Self-citationBest practices for data management and sharing in experimental biomedical research
Cunha‐Oliveira, T., Ioannidis, J. P. A., & Oliveira, P. J. (2024) · Physiological Reviews · doi:10.1152/physrev.00043.2023 · Cited by 57
Self-citationIn defense of quantitative metrics in researcher assessments
Ioannidis, J. P. A., & Maniadis, Z. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002408 · Cited by 41
“Bibliometrics should encompass indicators of best research practices (e.g., frequency of data sharing, code sharing, protocol registration, and replications) as a free, publicly available resource covering all the open-access literature [12].”
BackgroundSelf-citationMalički, M., Aalbersberg, I. J., Bouter, L. M., Mulligan, A., & ter Riet, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0270054 · Cited by 37
“Data sharing practices across sciences have not been systematically explored, but recent estimates indicated that data sharing was mentioned in 15% of biomedical [19], and in only 2% of psychological articles [20].”
BackgroundCadwallader, L., & Hrynaszkiewicz, I. (2022) · PeerJ · doi:10.7717/peerj.13933 · Cited by 30
“PLOS journals have higher code sharing rates, with 41% of PLOS Computational Biology article sharing code in 2019 (Serghiou, 2021).”
BackgroundDental Research Data Availability and Quality According to the FAIR Principles
Uribe, S. E., Sofi-Mahmudi, A., Raittio, E., Maldupa, I., & Vilne, B. (2022) · Journal of Dental Research · doi:10.1177/00220345221101321 · Cited by 27
“A survey conducted by Spallek et al. (2019), all 42 dental researchers responded that data sharing should be promoted and facilitated.”
BackgroundHamilton, D. G., Page, M. J., Finch, S. J., Everitt, S., & Fidler, F. M. (2022) · BMC Medicine · doi:10.1186/s12916-022-02644-2 · Cited by 51
“A notion that is supported by a recent and large study which observed an increase in data sharing declarations among papers available in PubMed Central between 2019 and 2020 [25].”
ResultMenke, J., Eckmann, P., Ozyurt, I. B., Roelandse, M., Anderson, N., Grethe, J., Gamst, A., & Bandrowski, A. (2022) · Journal of Medical Internet Research · doi:10.2196/37324 · Cited by 25
Cobey, K. D., Fehlmann, C. A., Franco, M. C., Ayala, A. P., Sikora, L., Rice, D. B., Xu, C., Ioannidis, J. P. A., Lalu, M. M., Ménard, A., et al. (2023) · eLife · doi:10.7554/elife.78518 · Cited by 26
“A recent evaluation of 349 randomly selected research articles from the biomedical literature published in 2015- 2018 (Serghiou et al., 2021) found that 33 (10%) included a reproducibility component in their research (e.”
Self-citationBaždarić, K., Vrkić, I., Arh, E., Mavrinac, M., Marković, M. G., Bilić-Zulle, L., Stojanovski, J., & Malički, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0244529 · Cited by 28
“Recent estimates show that data sharing was mentioned in only of 15% biomedical [16], and in only 2% of psychological articles [17].”
BackgroundCatalá-López, F., Ridao-López, M., Tejedor-Romero, L., Caulley, L., Hutton, B., Husereau, D., Alonso‐Arroyo, A., Bernal‐Delgado, E., Drummond, M., & Moher, D. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.10.024 · Cited by 21
Leonhardt, H., Wesemeyer, M., Eder, A., Hüttel, S., Lakes, T., Schaak, H., Seifert, S., & Wolff, S. (2024) · Ecological Indicators · doi:10.1016/j.ecolind.2024.112709 · Cited by 30
de Ridder, J. (2022) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2022.02.003 · Cited by 26
Transparency, bias, and reproducibility across science: a meta-research view
Ioannidis, J. P. A. (2024) · Journal of Clinical Investigation · doi:10.1172/jci181923 · Cited by 26
“There is more good news More sharing is happening over time across biomedicine, as we have documented in a large-scale evaluation of the entire open-access PubMed Central repository (43).”
Self-citationNaaman, K., Grant, S., Kianersi, S., Supplee, L. H., Henschel, B., & Mayo-Wilson, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.221093 · Cited by 30
“in databases such as PubMed), and support automated surveillance and meta-research concerning TOP [104].”
Coccia, M. (2024) · Contributions to management science · doi:10.1007/978-3-031-58359-9_5 · Cited by 24
What meta‐research has taught us about research and changes to research practices
Ioannidis, J. P. A. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12666 · Cited by 25
Self-citationTowards sharing tools and artefacts for reusable simulations in healthcare
Monks, T., Harper, A. L., & Mustafee, N. (2024) · Journal of Simulation · doi:10.1080/17477778.2024.2347882 · Cited by 18
“The practice of identifying code sharing from papers is laborious; across the biomedical literature, Serghiou et al. (2021) evaluated over 2.7 million open access articles using automated text extraction of transparency indicators, finding that 1% of papers had a code sharing statement.”
InfluentialLessons learnt from registration of biomedical research
Serghiou, S., Axfors, C., & Ioannidis, J. P. A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01499-0 · Cited by 19
Self-citationReflections on Preregistration: Core Criteria, Badges, Complementary Workflows
Thibault, R. T., Pennington, C. R., & Munafo, M. R. (2023) · Journal of Trial and Error · doi:10.36850/mr6 · Cited by 26
A meta-epidemiological assessment of transparency indicators of infectious disease models
Zavalis, E. A., & Ioannidis, J. P. A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275380 · Cited by 19
“[21] across the entire biomedical literature and has a positive predictive value (PPV) of 88.”
Self-citationHeinl, C., Scholman-Végh, A. M. D., Mellor, D. T., Schönfelder, G., Strech, D., Chamuleau, S. A. J., & Bert, B. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac016 · Cited by 18
“Despite the advantages for the global improvement of scientific progress, the uptake of preregistration in animal research is still slow (16, 17).”
BackgroundReplacing bar graphs of continuous data with more informative graphics: are we making progress?
Riedel, N., Schulz, R., Kazezian, V., & Weissgerber, T. L. (2022) · Clinical Science · doi:10.1042/cs20220287 · Cited by 19
“[20] that combines PMC articles with additional metadata (e.”
BackgroundSelf-citationOsborne, C., & Norris, E. (2022) · Cogent Psychology · doi:10.1080/23311908.2022.2066304 · Cited by 15
“Serghiou et al. (2021) assessed 2,751,420 PubMed Central open access publications from a range of research fields between 1959–2020 (e.g., the biological sciences, engineering, and the social sciences), and found 5.7% were pre-registered in 2020.”
BackgroundInfluentialPownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z. M., Grinschgl, S., Elsherif, M. M., Gilligan-Lee, K. A., Oliveira, C. M., et al. (2022) · doi:10.31222/osf.io/9e526 · Cited by 16
Toward practical transparent verifiable and long-term reproducible research using Guix
Vallet, N., Michonneau, D., & Tournier, S. (2022) · Scientific Data · doi:10.1038/s41597-022-01720-9 · Cited by 26
Research transparency in dental research: A programmatic analysis
Raittio, E., Sofi‐Mahmudi, A., & Uribe, S. E. (2023) · European Journal of Oral Sciences · doi:10.1111/eos.12908 · Cited by 15
“%) were less common in dental articles than in biomedical research articles (8.9% and 1.2%, respectively) [4].”
InfluentialTransparency of COVID-19-Related Research in Dental Journals
Sofi-Mahmudi, A., & Raittio, E. (2022) · Frontiers in Oral Health · doi:10.3389/froh.2022.871033 · Cited by 15
“Transparency indicators from the available full-texts were extracted with the rtransparent package [3].”
BackgroundMethodInfluentialTzanetakis, G. N., & Koletsi, D. (2021) · International Endodontic Journal · doi:10.1111/iej.13573 · Cited by 15
Hardwicke, T. E., Thibault, R. T., Clarke, B., Moodie, N., Crüwell, S., Schiavone, S. R., Handcock, S. A., Nghiem, K. A., Mody, F., Eerola, T., et al. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241283477 · Cited by 37
Analytical code sharing practices in biomedical research
Sharma, N. K., Ayyala, R., Deshpande, D., Patel, Y. M., Munteanu, V., Ciorbă, D., Bostan, V., Fiscutean, A., Vahed, M., Sarkar, A. P., et al. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2066 · Cited by 17
“By contrast, the limited availability of code and associated research products poses challenges to reproducibility across scienti fi c disciplines ( Serghiou et al., 2021 ; Baker, 2016 ; Hutson, 2018 ; Barnes, 2010 ).”
BackgroundStakeholders’ views on an institutional dashboard with metrics for responsible research
Haven, T. L., Holst, M., & Strech, D. (2022) · PLoS ONE · doi:10.1371/journal.pone.0269492 · Cited by 15
Institutional dashboards on clinical trial transparency for University Medical Centers: A case study
Franzen, D., Carlisle, B. G., Salholz‐Hillel, M., Riedel, N., & Strech, D. (2023) · PLoS Medicine · doi:10.1371/journal.pmed.1004175 · Cited by 15
Self-citationOpen Science Standards at Journals that Inform Evidence-Based Policy
Grant, S., Mayo‐Wilson, E., Kianersi, S., Naaman, K., & Henschel, B. (2023) · Prevention Science · doi:10.1007/s11121-023-01543-z · Cited by 22
“Structured data elements in manuscript submission systems and article templates also enable automatic checking during the manuscript submission and article production process, which could promote policy adherence (Serghiou et al., 2021).”
BackgroundOpen science practices for eating disorders research
Burke, N. L., Frank, G., Hilbert, A., Hildebrandt, T., Klump, K. L., Thomas, J. J., Wade, T. D., Walsh, B. T., Wang, S. B., & Weissman, R. S. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23607 · Cited by 13
How do we increase the trustworthiness of medical publications?
Mol, B. W., & Ioannidis, J. P. A. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.02.023 · Cited by 12
Self-citationHamilton, D. G., Fraser, H. S., Fidler, F. M., McDonald, S. J., Rowhani-Farid, A., Hong, K., & Page, M. J. (2021) · F1000Research · doi:10.12688/f1000research.53874.2 · Cited by 10
Three-stage publishing to support evidence-based management practice
Marin-Garcia, J. A. (2021) · WPOM - Working Papers on Operations Management · doi:10.4995/wpom.11755 · Cited by 14
“Estas tres características son escasamente cumplidas en la investigación reciente de áreas como la biomedicina (Serghiou et al., 2021) o la psicología (Hardwicke et al., 2021).”
BackgroundMethodInfluentialMetaresearch study highlights the gender gap in randomized controlled trials in dentistry
Prado, M. C., Dotto, L., Agostini, B. A., & Sarkis-Onofre, R. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.005 · Cited by 10
Biomedical supervisors’ role modeling of open science practices
Haven, T. L., Abunijela, S., & Hildebrand, N. (2023) · eLife · doi:10.7554/elife.83484 · Cited by 14
Peer review before trial conduct could increase research value and reduce waste
Siebert, M., Naudet, F., & Ioannidis, J. P. A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.05.024 · Cited by 10
Self-citationSilva, F. M., Adegboye, A. R. A., Lachat, C. K., Curioni, C. C., Gomes, F. D. S., Collins, G. S., Kac, G., de Beyer, J. A., Cook, J. A., Ismail, L. C., et al. (2023) · JMIR Research Protocols · doi:10.2196/43537 · Cited by 10
“Similarly, an automated assessment of all PubMed Central articles estimated that in 2020, overall 15% of articles had a data-sharing statement and 3% had a code-sharing statement [34].”
MethodCode-sharing policies are associated with increased reproducibility potential of ecological findings
Sánchez‐Tójar, A., Bezine, A., Purgar, M., & Čulina, A. (2025) · Peer Community Journal · doi:10.24072/pcjournal.541 · Cited by 13
Persistent gender disparity in leading dental publications across 4 decades: an observational study
Moreno, L. B., Franco, M. C., Karam, S. A., van de Sande, F. H., & Montagner, A. F. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111386 · Cited by 10
Workflow for detecting biomedical articles with underlying open and restricted-access datasets
Iarkaeva, A., Nachev, V., & Bobrov, E. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302787 · Cited by 9
“In the largest analysis so far conducted, Serghiou et al. [37] applied ODDPub to 2.75 million open access articles in PubMed Central (from 1959 till 2020, with the largest numbers in most recent years).”
InfluentialData sharing and transparency indicators in published RCTs in Oral Health between 2017 and 2023
Mikelis, F., Karamalaki, D., Mikeli, A., Tzanetakis, G. N., & Koletsi, D. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105263 · Cited by 8
Are open science practices in dentistry associated with higher Altmetric scores and citation rates?
Cenci, J., Mendes, F. M., Bouter, L. M., Pereira‐Cenci, T., Acosta, C. D. P., Brondani, B., Moher, D., Huysmans, M. C. D. N. J. M., & Cenci, M. S. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105393 · Cited by 9
Improving transparency in clinical trial reporting
Farrell, A., & Editors, O. B. O. T. P. M. (2025) · PLoS Medicine · doi:10.1371/journal.pmed.1004588 · Cited by 8
Reproducibility failure in biomedical research: problems and solutions
Haven, T. L., & Ioannidis, J. P. A. (2026) · Annual Review of Medicine · doi:10.1146/annurev-med-050124-050859 · Cited by 9
Self-citationHansford, H. J., Cashin, A. G., Bagg, M. K., Wewege, M. A., Ferraro, M. C., Kianersi, S., Mayo-Wilson, E., Grant, S., Toomey, E. C., Skinner, I., et al. (2022) · Sports Medicine - Open · doi:10.1186/s40798-022-00496-x · Cited by 14
AI and Open Science: Implications and Library Practice Recommendations
Helregel, N. (2025) · Library trends · doi:10.1353/lib.2025.a961191 · Cited by 7
Provenance and Funding of Extremely Cited Biomedical Articles Published Between 2003 and 2024
Ioannidis, J. P. A. (2025) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.3045 · Cited by 7
Self-citationTen simple rules for starting FAIR discussions in your community
Belliard, F., Maineri, A., Plomp, E., Padilla, A. F. R., Sun, J., & Jeddi, M. Z. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011668 · Cited by 8
Schueller, S., Mikelis, F., Eliades, T., & Koletsi, D. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae064 · Cited by 8
“Our findings on code or script sharing are particularly concerning, however, they align with broader trends in biomedicine [10].”
A comprehensive analysis of code and data availability in biomedical researchPreprint
Deshpande, D., Sarkar, A. P., Guo, R., Moore, A., Darci-Maher, N., & Mangul, S. (2021) · doi:10.31219/osf.io/uz7m5 · Cited by 6
Bannach‐Brown, A., Rackoll, T., Macleod, M. R., & McCann, S. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02524-2 · Cited by 8
“Although the statement “ data available upon request” is now commonly regarded as not fulfilling the criteria for open data, external links to datasets or protocols may break over time [20, 22].”
Pamporis, K., Bougioukas, K. I., Karakasis, P., Papageorgiou, D., Zarifis, I., & Haidich, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.05.018 · Cited by 6
Sharing study materials in health and medical research
DeVito, N., Morton, C. E., Cashin, A. G., Richards, G. C., & Lee, H. (2022) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-111987 · Cited by 10
Assessing transparency practices in dental randomized controlled trials
Prado, M. C., Dotto, L., Agostini, B., & Sarkis-Onofre, R. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02316-0 · Cited by 6
“Our findings align with previous biomedical and dental research, which reported the presence of more articles with statement of conflict of interest and funding than data and code sharing and protocol registration [27–30].”
ResultEstimating the replicability of highly cited clinical research (2004–2018)
da Costa, G. G., Neves, K., & Amaral, O. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307145 · Cited by 8
Lila, M. A., Ryan, C. D., & Diekman, C. B. (2025) · Annual Review of Food Science and Technology · doi:10.1146/annurev-food-111523-121925 · Cited by 6
Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals
Zavalis, E. A., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2023) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiad130 · Cited by 7
Self-citationGötz, M., & O’Boyle, E. H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0742-730120230000041004 · Cited by 10
Sofi‐Mahmudi, A., Raittio, E. J., Khazaei, Y., Ashraf, J., Schwendicke, F., Uribe, S. E., & Moher, D. (2024) · PLoS ONE · doi:10.1371/journal.pone.0313991 · Cited by 7
“%) in detecting the data availability of assessed papers (30).”
Hamilton, D. G., Hong, K., Fraser, H. S., Rowhani-Farid, A., Fidler, F. M., & Page, M. J. (2023) · medRxiv · doi:10.1101/2023.03.22.23287607 · Cited by 5
“Despite the included meta-research studies following similar methodologies, we do note high statistical heterogeneity for both analyses, with influence analyses showing that the greatest contributors to between-study heterogeneity for declared data sharing were the high precision findings of Uribe et al [125] and Serg…”
BackgroundMethodResultInfluentialThe paradox of competition: How funding models could undermine the uptake of data sharing practices
Klebel, T., Bianchi, F., Ross‐Hellauer, T., & Squazzoni, F. (2025) · Research Policy · doi:10.1016/j.respol.2025.105340 · Cited by 4
Falciola, L., & Barbieri, M. (2023) · World Patent Information · doi:10.1016/j.wpi.2023.102251 · Cited by 4
Wang, S., Schneider, M. Y., Volcke, E. I. P., & Wu, D. (2025) · Water Research X · doi:10.1016/j.wroa.2025.100343 · Cited by 4
Hamilton, D. G., Page, M. J., Everitt, S., Fraser, H. S., & Fidler, F. M. (2024) · Accountability in Research · doi:10.1080/08989621.2024.2308606 · Cited by 5
“…in oncology (Hamilton et al. 2022; Walters et al. 2019), medicine in general (Hamilton et al. 2023) and most scientific disciplines more broadly (Serghiou et al. 2021), many surveys of researchers, as well as other key stakeholders (e.g., funders, publishers, research participants) have been…”
Bullock, G. S., Ward, P., Kluzek, S., Hughes, T., Shanley, E., Arundale, A. J. H., Ranson, C., Nimphius, S., Riley, R. D., Collins, P. G. S., et al. (2023) · British Journal of Sports Medicine · doi:10.1136/bjsports-2023-107225 · Cited by 6
Hardwicke, T. E., Thibault, R. T., Clarke, B., Moodie, N., Crüwell, S., Schiavone, S. R., Handcock, S. A., Nghiem, K. A., Mody, F., Eerola, T., et al. (2024) · doi:10.31234/osf.io/t2zs9 · Cited by 4
Park, J. R., Bae, I., Ryu, S., Kim, M., Bang, H., Won, J., & Lee, H. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101229 · Cited by 3
Reporting completeness of scoping reviews in orthodontic literature up to 2022. An empirical study
Mikelis, F., & Koletsi, D. (2023) · European Journal of Orthodontics · doi:10.1093/ejo/cjad022 · Cited by 4
Silva, F. M., Adegboye, A. R. A., Curioni, C. C., Gomes, F. D. S., Collins, G. S., Kac, G., Cook, J. A., Ismail, L. C., Page, M. J., Khandpur, N., et al. (2023) · medRxiv · doi:10.1101/2023.06.08.23291109 · Cited by 3
Research Transparency in 59 Disciplines of Clinical Medicine: A Meta-Research StudyPreprint
Sofi‐Mahmudi, A., Raittio, E. J., Uribe, S. E., Khademioore, S., Zeraatkar, D., Mbuagbaw, L. C. E., Bouter, L. M., & Robinson, K. A. (2024) · MetArXiv (OSF Preprints) · doi:10.1101/2024.04.08.24305416 · Cited by 4
“Key indicators of research transparency include data sharing, code sharing, disclosures of conflicts of interest, funding acknowledgments, and protocol registration (13).”
BackgroundMethodInfluentialSiddique, A. B., Shaw, B. P., Dwyer, J. T., Fields, D. A., Fontaine, K. R., Hand, D. J., Schekman, R. W., Alberts, J. R., Locher, J. L., & Allison, D. B. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00517-w · Cited by 3
“One promising initiative is the open-source, automated system proposed by Serghiou et al. (2021), which identifies five indicators of transparency: data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration.”
BackgroundPreregistration in Animal Research
Heinl, C., Schönfelder, G., & Bert, B. (2022) · Springer eBooks · doi:10.1007/978-3-030-99680-2_37 · Cited by 2
Conflict of Interest and Commitment and Research Integrity
Redman, B. K. (2023) · Springer eBooks · doi:10.1007/978-3-031-27111-3_6 · Cited by 4
Transparency of COVID-19-related research: A meta-research study
Sofi‐Mahmudi, A., Raittio, E. J., & Uribe, S. E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288406 · Cited by 4
Influencia de las redes sociales en la descarga de artículos de Radiología
de la Gala, D. H., Alcalá, E. S., Domènech-Ximenos, B., & Villar, C. G. (2024) · Radiología · doi:10.1016/j.rx.2024.08.002 · Cited by 2
Assessment of transparency indicators in space medicine
Bellomo, R. K., Zavalis, E. A., & Ioannidis, J. P. A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300701 · Cited by 2
“Open access is in fact another dimension of transparency [9].”
BackgroundMethodInfluentialSelf-citationOpen science practices and risks arising from misuse of biological researchPreprint
Smith, J. A., & Sandbrink, J. B. (2021) · doi:10.31222/osf.io/8afwn · Cited by 2
Do randomised clinical trials on dental caries adopt Open Science practices?
Acosta, C. D. P., Cenci, J., Brondani, B., Elagami, R. A., Pereira-Cenci, T., Cenci, M. S., Huysmans, M.-C. D. N. J. M., Raggio, D. P., Braga, M. M., & Mendes, F. M. (2024) · BMC Oral Health · doi:10.1186/s12903-024-05218-1 · Cited by 4
“In other articles, the authors only stated that the data could be made available on reasonable request, which is not counted as “current” data sharing in previous studies [20, 35].”
Teaching reproducible research for medical students and postgraduate pharmaceutical scientists
Meid, A. D. (2021) · BMC Research Notes · doi:10.1186/s13104-021-05862-8 · Cited by 2
“There is now consensus that the value of research can be enhanced by greater transparency and openness in the processes of research design, conduct, analysis, and reporting [1, 2].”
BackgroundLes revues SIC et les données de recherche: Une étude empirique
Schöpfel, J., & Kergosien, É. (2021) · Balisages · doi:10.35562/balisages.641 · Cited by 2
Karakitsos, P., & Mylonas, K. S. (2024) · International Journal of Cardiology · doi:10.1016/j.ijcard.2024.131987 · Cited by 1
Metadata
- Authors
- Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- SciTech Strategies (United States) (US)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Yale University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-03-01
- Volume
- 19
- Issue
- 3
- Pages
- e3001107
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC7951980
- OpenAlex
- W3097500482
- Cited by
- 139
- References
- 42
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 29 · 2023: 33 · 2022: 30 · 2021: 13
- FWCI
- 22.39
- Type
- journal article
- Language
- en
- Update notices
- new_version (2021-03-11)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (HHSN271201800033C)
- Laura and John Arnold Foundation
- Center of Excellence in Regulatory Science and Innovation (U01FD005938)
- School of Medicine, Stanford University
- Stanford Data Science Institute, Stanford University
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- research transparency, reproducibility, data sharing, code sharing, conflict of interest disclosure, funding disclosure, protocol registration, biomedical literature, PubMed Central, open access publications, temporal trends
- MeSH terms
- Scholarly Communication, Humans, Publications, Reproducibility of Results, Databases, Factual, Conflict of Interest, Disclosure, Information Dissemination, Biomedical Research, Open Access Publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.