Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017
Open accessThe DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
Currently, there is a growing interest in ensuring the transparency and reproducibility of the published scientific literature. According to a previous evaluation of 441 biomedical journals articles published in 2000-2014, the biomedical literature largely lacked transparency in important dimensions. Here, we surveyed a random sample of 149 biomedical articles published between 2015 and 2017 and determined the proportion reporting sources of public and/or private funding and conflicts of interests, sharing protocols and raw data, and undergoing rigorous independent replication and reproducibility checks. We also investigated what can be learned about reproducibility and transparency indicators from open access data provided on PubMed. The majority of the 149 studies disclosed some information regarding funding (103, 69.1% [95% confidence interval, 61.0% to 76.3%]) or conflicts of interest (97, 65.1% [56.8% to 72.6%]). Among the 104 articles with empirical data in which protocols or data sharing would be pertinent, 19 (18.3% [11.6% to 27.3%]) discussed publicly available data; only one (1.0% [0.1% to 6.0%]) included a link to a full study protocol. Among the 97 articles in which replication in studies with different data would be pertinent, there were five replication efforts (5.2% [1.9% to 12.2%]). Although clinical trial identification numbers and funding details were often provided on PubMed, only two of the articles without a full text article in PubMed Central that discussed publicly available data at the full text level also contained information related to data sharing on PubMed; none had a conflicts of interest statement on PubMed. Our evaluation suggests that although there have been improvements over the last few years in certain key indicators of reproducibility and transparency, opportunities exist to improve reproducible research practices across the biomedical literature and to make features related to reproducibility more readily visible in PubMed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Particular issues are presented by the use of published data, which can be unvalidated or incomplete due to selective reporting26,27.”
From Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency (2020) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the literature contains studies with questionable methodologies, missing or incomplete data, false or exaggerated effect measures, and many studies do not mention funding or potential conflicts of interest [4–8].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies indicate that, although many authors agree on the importance of these transparent practices, they are often neglected [6, 7].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [7]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citation1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationPromoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Self-citation
Research works printed but not linked: 3
- PLoS ONE. Data Availability. Available from: http://journals.plos.org/plosone/s/data-availability.
- BMC Research Notes. Aims and scope. https://bmcresnotes.biomedcentral.com/.
- D de Weerd-Wilson, 2017, How Elsevier is breaking down barriers to reproducibility, How Elsevier is breaking down barriers to reproducibility
Cited by
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Self-citationOpen science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540
Self-citationData sharing practices and data availability upon request differ across scientific disciplines
Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Astapova, A., Lukner, H., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00981-0 · Cited by 387
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
The citation advantage of linking publications to research data
Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., & McGillivray, B. (2020) · PLOS ONE · doi:10.1371/journal.pone.0230416 · Cited by 335
“It is known that, for example, the biomedical literature in PubMed has shown clear signs of improvement in the transparency and reproducibility of results over recent years, including sharing data [23].”
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
Self-citationHardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
“%) than social-science articles in general (31% and 15%, respectively; Hardwicke, Wallach, et al., 2020) but less likely than biomedical articles (69% and 65%, respectively; Wallach et al., 2018).”
BackgroundResultInfluentialSelf-citationReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Kourou, K., Exarchos, K. P., Papaloukas, C., Sakaloglou, P., Exarchos, T., & Fotiadis, D. I. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.006 · Cited by 165
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148
Assessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139
Self-citationApplying artificial intelligence for cancer immunotherapy
Xu, Z., Wang, X., Zeng, S., Ren, X., Yan, Y., & Gong, Z. (2021) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2021.02.007 · Cited by 126
“Second, the reproducibility of AI is difficult to evaluate due to the limited availability of code and datasets, which might be solved by encouraging code and data sharing across scientific disciplines 127 e 129 .”
BackgroundGenomic Features of Parthenogenetic Animals
Jaroň, K. S., Bast, J., Nowell, R. W., Ranallo-Benavidez, T. R., Robinson‐Rechavi, M., & Schwander, T. (2020) · Journal of Heredity · doi:10.1093/jhered/esaa031 · Cited by 124
“Another advantage of using the same methods for each species is that it diminishes the “researcher degrees of freedom” (Simmons et al. 2011; Gelman and Loken 2013; Wallach et al. 2018).”
The complexities of the diet-microbiome relationship: advances and perspectives
Leeming, E. R., Louca, P., Gibson, R., Menni, C., Spector, T. D., & Roy, C. I. L. (2021) · Genome Medicine · doi:10.1186/s13073-020-00813-7 · Cited by 123
“Platforms such as MGnify, a free-to-use platform for the assembly, analysis and archiving of microbial data have allowed for publicly available analysed datasets [67]; however, extensive action is required to populate these platforms, and differences between pipelines can also lead to variations in outcomes.”
BackgroundAn overview and update on the epidemiology of flavonoid intake and cardiovascular disease risk
Parmenter, B. H., Croft, K. D., Hodgson, J. M., Dalgaard, F., Bondonno, C. P., Lewis, J. R., Cassidy, A., Scalbert, A., & Bondonno, N. P. (2020) · Food & Function · doi:10.1039/d0fo01118e · Cited by 126
Who funded the research behind the Oxford–AstraZeneca COVID-19 vaccine?
Cross, S., Rho, Y., Reddy, H., Pepperrell, T., Rodgers, F., Osborne, R., Eni‐Olotu, A., Banerjee, R., Wimmer, S., & Keestra, S. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-007321 · Cited by 112
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“Replication studies remain uncommon in many fields (Hardwicke et al. 2019b, Iqbal et al. 2016, Makel et al. 2012, Sterling 1959, Wallach et al. 2018).”
Self-citationEULAR points to consider for the use of big data in rheumatic and musculoskeletal diseases
Gossec, L., Kedra, J., Servy, H., Pandit, A., Stones, S., Berenbaum, F., Finckh, A., Baraliakos, X., Stamm, T. A., Gomez-Cabrero, D., et al. (2020) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2019-215694 · Cited by 90
Did awarding badges increase data sharing inBMJ Open? A randomized controlled trial
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020) · Royal Society Open Science · doi:10.1098/rsos.191818 · Cited by 101
“A recent update to this study, however, has shown an increase of data sharing in biomedical research articles published between 2015 and 2017 where 18% of papers discuss data availability [9].”
BackgroundBradley, S. H., DeVito, N., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C., & Gill, P. J. (2020) · Journal of the Royal Society of Medicine · doi:10.1177/0141076820956799 · Cited by 112
“Interest disclosures for published research are frequently incomplete.(16,17) Conflict of interest recording and policies in institutions that host research are also poor, and in several cases journal editors, as well as researchers, have potential conflicts of interest.”
BackgroundHamilton, D. G., Hong, K., Fraser, H., Rowhani-Farid, A., Fidler, F., & Page, M. J. (2023) · BMJ · doi:10.1136/bmj-2023-075767 · Cited by 86
Helicobacter pylori infection and gastric cancer biology: tempering a double-edged sword
Mentis, A. A., Boziki, M. K., Grigoriadis, N., & Papavassiliou, A. G. (2019) · Cellular and Molecular Life Sciences · doi:10.1007/s00018-019-03044-1 · Cited by 88
Ten simple rules for carrying out and writing meta-analyses
Forero, D. A., López‐León, S., González‐Giraldo, Y., & Bagos, P. G. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006922 · Cited by 94
The War against Tuberculosis: A Review of Natural Compounds and Their Derivatives
Maiolini, M., Gause, S., Taylor, J., Steakin, T., Shipp, G. M., Lamichhane, P., Deshmukh, B., Shinde, V. M., Bishayee, A., & Deshmukh, R. R. (2020) · Molecules · doi:10.3390/molecules25133011 · Cited by 85
Littmann, M., Selig, K., Cohen-Lavi, L., Frank, Y., Hönigschmid, P., Kataka, E., Mösch, A., Qian, K., Ron, A., Schmid, S., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0139-8 · Cited by 70
Point estimates, Simpson’s paradox, and nonergodicity in biological sciences
Mangalam, M., & Kelty‐Stephen, D. G. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.017 · Cited by 78
López‐Nicolás, R., López-López, J. A., Rubio‐Aparicio, M., & Sánchez‐Meca, J. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01644-z · Cited by 80
Open science and public trust in science: Results from two studies
Rosman, T., Bošnjak, M., Silber, H., Koßmann, J., & Heycke, T. (2022) · Public Understanding of Science · doi:10.1177/09636625221100686 · Cited by 75
Vagliano, I., Chesnaye, N. C., Leopold, J. H., Jager, K. J., Abu-Hanna, A., & Schut, M. C. (2022) · Clinical Kidney Journal · doi:10.1093/ckj/sfac181 · Cited by 59
“Reproducible research has become a pressing issue across many scientific disciplines, and sharing data and code is key [47, 51, 52].”
BackgroundToward unrestricted use of public genomic data
Amann, R., Baichoo, S., Blencowe, B. J., Bork, P., Borodovsky, M. Y., Brooksbank, C., Chain, P., Colwell, R. R., Daffonchio, D. G., Danchin, A., et al. (2019) · Science · doi:10.1126/science.aaw1280 · Cited by 56
Self-citationBarriers to Full Participation in the Open Science Life Cycle among Early Career Researchers
Gownaris, N. J., Vermeir, K., Bittner, M.-I., Gunawardena, L., Kaur‐Ghumaan, S., Lepenies, R., Ntsomboh-Ntsefong, G., & Zakari, I. S. (2022) · Data Science Journal · doi:10.5334/dsj-2022-002 · Cited by 81
Liu, Y., Acharya, U. R., & Tan, J. H. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108571 · Cited by 93
LitSense: making sense of biomedical literature at sentence level
Allot, A., Chen, Q., Kim, S., Vera Alvarez, R., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz289 · Cited by 56
Biosecurity in an age of open science
Smith, J. A., & Sandbrink, J. B. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001600 · Cited by 70
Wallach, J. D., Serghiou, S., Chu, L., Egilman, A. C., Vasiliou, V., Ross, J. S., & Ioannidis, J. P. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0914-6 · Cited by 57
Self-citationPage, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 61
“to improve the reporting of methods and make the underlying materials, such as data and analytic code, publicly accessible [8, 9].”
MethodPage, M. J., Nguyen, P., Hamilton, D. G., Haddaway, N., Kanukula, R., Moher, D., & McKenzie, J. E. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.03.003 · Cited by 59
Cashin, A. G., Bagg, M. K., Richards, G. C., Toomey, E. C., McAuley, J. H., & Lee, H. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111296 · Cited by 65
Pitfalls in Cancer Biomarker Discovery and Validation with Emphasis on Circulating Tumor DNA
Ren, A. H., Fiala, C., Diamandis, E. P., & Kulasingam, V. (2020) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-20-0074 · Cited by 58
Machine Learning–Based Prediction Models for Delirium: A Systematic Review and Meta-Analysis
Xie, Q., Wang, X., Pei, J., Wu, Y., Guo, Q., Su, Y., Yan, H., Nan, R., Chen, H., & Dou, X. (2022) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2022.06.020 · Cited by 58
Lu, C., Ke, L., Li, J., Zhao, H., Lu, T., Mentis, A. A., Wang, Y., Wang, Z., Polissiou, M. G., Tang, L., et al. (2021) · Phytomedicine · doi:10.1016/j.phymed.2021.153699 · Cited by 51
Open laboratory notebooks: good for science, good for society, good for scientists
Schapira, M., & Harding, R. (2019) · F1000Research · doi:10.12688/f1000research.17710.2 · Cited by 47
“Second, open lab notebooks should include detailed protocols that can be reproduced, which is often not the case in peer-reviewed publications 5,6 .”
Stewart, J. E., Lü, J., Goudie, A., Bennamoun, M., Sprivulis, P., Sanfilippo, F. M., & Dwivedi, G. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252612 · Cited by 53
Reproducible research practices and transparency across linguistics
Bochyńska, A., Keeble, L., Halfacre, C. A. E., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011239 · Cited by 54
“…articles with primary data analyses reported available materials (pre-RC: 14.8%; post-RC: 24.3%), which is slightly higher than estimates from sociology (11%; Hardwicke et al., 2020) and psychology (14%; Hardwicke et al., 2022), but lower than estimates from biomedicine (33%; Wallach et al., 2018).”
InfluentialData management and sharing: Practices and perceptions of psychology researchers
Borghi, J., & Gulick, A. E. V. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252047 · Cited by 51
Estimating the Replicability of Sports and Exercise Science Research
Murphy, J., Caldwell, A. R., Mesquida, C., Ladell, A. J. M., Martínez, A. E., Tual, A., Denys, A. T., Cameron, B., Hooren, B. V., Parr, B., et al. (2025) · Sports Medicine · doi:10.1007/s40279-025-02201-w · Cited by 59
“The level of data sharing in sports and exercise science appears to be considerably lower than in other fields [89, 90].”
Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020) · Trials · doi:10.1186/s13063-019-3944-9 · Cited by 38
Transparency Assessment of COVID-19 Models
Jalali, M. S., DiGennaro, C., & Sridhar, D. (2020) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30447-2 · Cited by 37
Catalá-López, F., Caulley, L., Ridao-López, M., Hutton, B., Husereau, D., Drummond, M., Alonso‐Arroyo, A., Pardo-Fernández, M., Bernal‐Delgado, E., Meneu, R., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034463 · Cited by 41
A multi-omics digital research object for the genetics of sleep regulation
Jan, M., Gobet, N., Diessler, S., Franken, P., & Xenarios, I. (2019) · Scientific Data · doi:10.1038/s41597-019-0171-x · Cited by 36
Pre-Clinical Common Data Elements for Traumatic Brain Injury Research: Progress and Use Cases
LaPlaca, M. C., Huie, J. R., Alam, H. B., Bachstetter, A. D., Bayır, H., Bellgowan, P. S. F., Cummings, D. M., Dixon, C. E., Ferguson, A. R., Ferland‐Beckham, C., et al. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2020.7328 · Cited by 34
Machine learning‐based prediction models for pressure injury: A systematic review and meta‐analysis
Pei, J., Guo, X., Tao, H., Wei, Y., Zhang, H., Ma, Y., & Han, L. (2023) · International Wound Journal · doi:10.1111/iwj.14280 · Cited by 41
“Given that reproducibility of research has become a growing concern across many scientific domains, 48-50 addressing these challenges of transparency and reproducibility are important in delivering health benefits from ML. Interestingly, many models only use a small subset of the wealth of…”
Christensen, M., Zellers, J. A., Kjær, I. L., Silbernagel, K. G., & Rathleff, M. S. (2020) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2020.9463 · Cited by 37
Analysis of practices to promote reproducibility and transparency in anaesthesiology research
Okonya, O., Rorah, D., Tritz, D., Umberham, B. A., Wiley, M., & Vassar, M. B. (2020) · British Journal of Anaesthesia · doi:10.1016/j.bja.2020.03.035 · Cited by 30
Vazquez, E., Gouraud, H., Naudet, F., Gross, C. P., Krumholz, H. M., Ross, J. S., & Wallach, J. D. (2021) · Clinical Trials · doi:10.1177/17407745211038524 · Cited by 33
Self-citationDental Research Data Availability and Quality According to the FAIR Principles
Uribe, S. E., Sofi-Mahmudi, A., Raittio, E., Maldupa, I., & Vilne, B. (2022) · Journal of Dental Research · doi:10.1177/00220345221101321 · Cited by 27
Algorithmic bias in social research: A meta-analysis
Thiem, A., Mkrtchyan, L., Haesebrouck, T., & Sánchez, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233625 · Cited by 29
Ten simple rules for providing effective bioinformatics research support
Kumuthini, J., Chimenti, M. S., Nahnsen, S., Peltzer, A., Meraba, R. L., McFadyen, R., Wells, G., Taylor, D. M., Maienschein‐Cline, M., Li, J., et al. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007531 · Cited by 32
An evaluation of the practice of transparency and reproducibility in addiction medicine literature
Adewumi, M. T., Vo, N. H., Tritz, D., Beaman, J., & Vassar, M. B. (2020) · Addictive Behaviors · doi:10.1016/j.addbeh.2020.106560 · Cited by 27
Hamilton, D. G., Page, M. J., Finch, S. J., Everitt, S., & Fidler, F. M. (2022) · BMC Medicine · doi:10.1186/s12916-022-02644-2 · Cited by 51
“Furthermore, the proportion of articles that included a formalised data availability statement in our study (65, 21%), which is also now a requirement for articles reporting the results of clinical trials [51], is also consistent with prior research in medicine such as Wallach and colleagues [40] who observed a substa…”
MethodResultMiner, A. S., Stewart, S. A., Halley, M. C., Nelson, L. K., & Linos, E. (2023) · Big Data & Society · doi:10.1177/20539517221149106 · Cited by 30
“This matters because concerns have been raised around reproducibility in social science (Tannenbaum et al., 2019; Wallach et al., 2018).”
BackgroundBorg, D. N., Bon, J. J., Sainani, K. L., Baguley, B. J., Tierney, N. J., & Drovandi, C. (2020) · Sports Medicine · doi:10.1007/s40279-020-01298-5 · Cited by 31
Reproducible and transparent research practices in published neurology research
Rauh, S. L., Torgerson, T., Johnson, A., Pollard, J., Tritz, D., & Vassar, M. B. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-0091-5 · Cited by 27
O’Shea, R. J., Sharkey, A. R., Cook, G., & Goh, V. (2021) · European Radiology · doi:10.1007/s00330-021-07881-2 · Cited by 31
Cobey, K. D., Fehlmann, C. A., Franco, M. C., Ayala, A. P., Sikora, L., Rice, D. B., Xu, C., Ioannidis, J. P. A., Lalu, M. M., Ménard, A., et al. (2023) · eLife · doi:10.7554/elife.78518 · Cited by 29
“Random samples of biomedical papers with empirical data published in 2015–2018 have shown that reproducibility concepts are not uncommon (Wallach et al., 2018).”
BackgroundSelf-citationYeung, A. W. K., Robertson, M., Uecker, A. M., Fox, P. T., & Eickhoff, S. B. (2022) · Human Brain Mapping · doi:10.1002/hbm.26177 · Cited by 26
Walters, C. G., Harter, Z. J., Wayant, C., Vo, N. H., Warren, M., Chronister, J., Tritz, D., & Vassar, M. B. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033962 · Cited by 24
Increase value and reduce waste in research on psychological therapies
Cristea, I. A., & Naudet, F. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103479 · Cited by 22
Catalá-López, F., Ridao-López, M., Tejedor-Romero, L., Caulley, L., Hutton, B., Husereau, D., Alonso‐Arroyo, A., Bernal‐Delgado, E., Drummond, M., & Moher, D. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.10.024 · Cited by 21
Ricolfi, L., Vendl, C., Bräunig, J., Taylor, M. D., Hesselson, D., Neely, G. G., Lagisz, M., & Nakagawa, S. (2024) · Environment International · doi:10.1016/j.envint.2024.108860 · Cited by 30
Data sharing practices in randomized trials of addiction interventions
Vassar, M. B., Jellison, S., Wendelbo, H., & Wayant, C. (2019) · Addictive Behaviors · doi:10.1016/j.addbeh.2019.106193 · Cited by 22
Open Science Practices in Gambling Research Publications (2016–2019): A Scoping Review
Louderback, E. R., Gainsbury, S., Heirene, R. M., Amichia, K., Grossman, A., Bernhard, B. J., & LaPlante, D. A. (2022) · Journal of Gambling Studies · doi:10.1007/s10899-022-10120-y · Cited by 22
An analysis of key indicators of reproducibility in radiology
Wright, B., Vo, N. H., Nolan, J., Johnson, A., Braaten, T., Tritz, D., & Vassar, M. B. (2020) · Insights into Imaging · doi:10.1186/s13244-020-00870-x · Cited by 19
Buvat, I., & Orlhac, F. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.120.261586 · Cited by 25
Pathways to breast cancer screening artificial intelligence algorithm validation
Lee, C. I., Houssami, N., Elmore, J. G., & Buist, D. S. M. (2019) · The Breast · doi:10.1016/j.breast.2019.09.005 · Cited by 26
Vibration of effects in epidemiologic studies of alcohol consumption and breast cancer risk
Chu, L., Ioannidis, J. P. A., Egilman, A. C., Vasiliou, V., Ross, J. S., & Wallach, J. D. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz271 · Cited by 23
Self-citationMeyer, A., Faverjon, C., Hostens, M., Stegeman, A., & Cameron, A. R. (2021) · BMC Veterinary Research · doi:10.1186/s12917-021-02971-1 · Cited by 20
Effects of psychedelics on neurogenesis and broader neuroplasticity: a systematic review
da Cruz, R. V. L., Leão, R. N., & Moulin, T. C. (2024) · Molecular Medicine · doi:10.1186/s10020-024-01013-4 · Cited by 22
Availability of Research Data in High-Impact Addiction Journals with Data Sharing Policies
Gorman, D. M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00203-7 · Cited by 20
Héroux, M. E., Butler, A. A., Cashin, A. G., McCaughey, E. J., Affleck, A. J., Green, M. A., Cartwright, A., Jones, M. D., Kiely, K. M., van Schooten, K. S., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-060976 · Cited by 21
Gardener, A. D., Hick, E. J., Jacklin, C., Tan, G., Cashin, A. G., Lee, H., Nunan, D., Toomey, E. C., & Richards, G. C. (2022) · JRSM Open · doi:10.1177/20542704221132139 · Cited by 23
Open science failed to penetrate academic hiring practices: a cross-sectional study
Khan, H., Almoli, E., Franco, M. C., & Moher, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.003 · Cited by 20
Is artificial intelligence prepared for the 24-h shifts in the ICU?
Gonzalez, F. A., Santonocito, C., Lamas, T., Costa, P. H. P., Vieira, S. M., Ferreira, H. A., & Sanfilippo, F. (2024) · Anaesthesia Critical Care & Pain Medicine · doi:10.1016/j.accpm.2024.101431 · Cited by 39
Rousi, A. M. (2022) · Journal of Librarianship and Information Science · doi:10.1177/09610006221093049 · Cited by 18
“In addition, data sharing in biomedical research has been studied using random samples of articles (Wallach et al., 2018).”
BackgroundLuong, J. C., Press, D. M., & Holl, K. D. (2023) · Biological Conservation · doi:10.1016/j.biocon.2023.109956 · Cited by 19
Perspectives on scientific error
van Ravenzwaaij, D., Bakker, M., Heesen, R., Romero, F., van Dongen, N. N. N., Crüwell, S., Field, S. M., Held, L., Munafo, M. R., Pittelkow, M.-M., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.230448 · Cited by 33
“This is consistent with the estimate of sharing analysis scripts in biomedicine [64], indicating that this is not an isolated issue.”
ResultSandbank, M. P., Bottema‐Beutel, K., Syu, Y.-C., Caldwell, N., Feldman, J. I., & Woynaroski, T. G. (2024) · Autism · doi:10.1177/13623613241231624 · Cited by 16
“Evidence suggests that the use of open science practices has increased in recent years (though there is still room for improvement; Wallach et al., 2018) and that increased use of prospective trial registration may reduce the potential inflation of effects caused by reporting and publication biases.”
BackgroundEvaluation of indicators supporting reproducibility and transparency within cardiology literature
Anderson, J. M., Wright, B., Rauh, S. L., Tritz, D., Horn, J. G., Parker, I., Bergeron, D., Cook, S., & Vassar, M. B. (2020) · Heart · doi:10.1136/heartjnl-2020-316519 · Cited by 19
Hadad, S., & Aharony, N. (2022) · Online Information Review · doi:10.1108/oir-01-2022-0014 · Cited by 19
Assessment of transparent and reproducible research practices in the psychiatry literature
Sherry, C. E., Pollard, J., Tritz, D., Carr, B., Pierce, A. M., & Vassar, M. B. (2020) · General Psychiatry · doi:10.1136/gpsych-2019-100149 · Cited by 18
Bullock, G. S., Ward, P., Impellizzeri, F. M., Kluzek, S., Hughes, T., Hillman, C., Waterman, B. R., Danelson, K., Henry, K., Barr, E., et al. (2023) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2023.12016 · Cited by 15
Research transparency in dental research: A programmatic analysis
Raittio, E., Sofi‐Mahmudi, A., & Uribe, S. E. (2023) · European Journal of Oral Sciences · doi:10.1111/eos.12908 · Cited by 15
“However, investigation of biomedical research articles has shown that, in terms of these five transparency indicators, there are only small differences between articles accessible and inaccessible via the EPMC [40, 41].”
Addressing selective reporting of experiments through predefined exclusion criteria
Neves, K., & Amaral, O. B. (2020) · eLife · doi:10.7554/elife.56626 · Cited by 15
Norris, E., He, Y., Loh, R., West, R., Michie, S., & Ward, K. D. (2021) · The Journal of Smoking Cessation · doi:10.1155/2021/6694386 · Cited by 12
“Metascience studies have assessed reproducibility and transparency across other disciplines, including 250 studies in social sciences [20], 149 studies in biomedicine [21], and 480 studies across two journals in biostatistics [22], all with no restrictions on study designs.”
BackgroundInfluentialOpen Data Policies among Library and Information Science Journals
Jackson, B. (2021) · Publications · doi:10.3390/publications9020025 · Cited by 21
“Research has examined data availability rates in journals from a variety of disciplines, including biology, chemistry, mathematics, and physics [5], biomedical disciplines [6], neurology [7], ecology and evolution [8], political science [9], and addiction studies [10], among others.”
BackgroundChui, K. T., Lytras, M. D., Visvizi, A., & Sarirete, A. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-822060-3.00015-2 · Cited by 22
Assessing Open Science practices in physical activity behaviour change intervention evaluations
Norris, E., Sulevani, I., Finnerty, A. N., & Castro, O. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2021-001282 · Cited by 14
Borg, D. N., Bon, J. J., Sainani, K. L., Baguley, B. J., Tierney, N. J., & Drovandi, C. (2020) · SportRxiv (OSF Preprints) · doi:10.31236/osf.io/ftdgj · Cited by 16
Zheng, E., Fang, Z., & Fu, H. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2023.103640 · Cited by 15
Metadata
- Authors
- Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A.
- Affiliations
- Yale University (US)
- SciTech Strategies (United States) (US)
- Stanford Medicine (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-11-20
- Volume
- 16
- Issue
- 11
- Pages
- e2006930
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC6245499
- OpenAlex
- W2901231347
- Cited by
- 269
- References
- 38
- Citations by year
- 2026: 21 · 2025: 30 · 2024: 30 · 2023: 31 · 2022: 33 · 2021: 42 · 2020: 52 · 2019: 27 · 2018: 2
- FWCI
- 43.45
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation
- Laura and John Arnold Foundation
- National Institutes of Health
- National Institute on Drug Abuse (HHSN271201700041C)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- reproducibility, research transparency, open data, conflicts of interest, funding disclosure, data sharing, replication study, PubMed, biomedical literature
- MeSH terms
- Humans, Publications, Reproducibility of Results, Conflict of Interest, Access to Information, Disclosure, Information Dissemination, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.