Reproducible Research Practices and Transparency across the Biomedical Literature
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
There is a growing movement to encourage reproducibility and transparency practices in the scientific community, including public access to raw data and protocols, the conduct of replication studies, systematic integration of evidence in systematic reviews, and the documentation of funding and potential conflicts of interest. In this survey, we assessed the current status of reproducibility and transparency addressing these indicators in a random sample of 441 biomedical journal articles published in 2000-2014. Only one study provided a full protocol and none made all raw data directly available. Replication studies were rare (n = 4), and only 16 studies had their data included in a subsequent systematic review or meta-analysis. The majority of studies did not mention anything about funding or conflicts of interest. The percentage of articles with no statement of conflict decreased substantially between 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles reporting statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0% in 2014) increased. Articles published in journals in the clinical medicine category versus other fields were almost twice as likely to not include any information on funding and to have private funding. This study provides baseline data to compare future progress in improving these indicators in the scientific literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the absence of stringent community norms or regulations, scientific claims are regularly published without public release of the research data upon which they are based [7,9,14].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the literature contains studies with questionable methodologies, missing or incomplete data, false or exaggerated effect measures, and many studies do not mention funding or potential conflicts of interest [4–8].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, assessment of analytic reproducibility is typically precluded by poor access to research data [7–14].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Replication validity of genetic association studies
Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., & Contopoulos-Ioannidis, D. G. (2001) · Nature Genetics · doi:10.1038/ng749 · Cited by 1,832
Self-citationScope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review
Bekelman, J. E., Li, Y., & Gross, C. P. (2003) · JAMA · doi:10.1001/jama.289.4.454 · Cited by 1,816
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationReproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationAnalysis of DNA methylation in cancer: location revisited
Koch, A., Joosten, S. C., Feng, Z., de Ruijter, T. C., Draht, M. X., Melotte, V., Smits, K. M., Veeck, J., Herman, J. G., Van Neste, L., et al. (2018) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-018-0004-4 · Cited by 710
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700
“The result is that a tiny minority of data from published articles are shared [7,20].”
BackgroundA solution to minimum sample size for regressions
Jenkins, D. G., & Quintana-Ascencio, P. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229345 · Cited by 665
“Small N has been discussed as one part of a larger reproducibility problem, where recommended solutions include registered studies, conflict of interest statements, and publication of data [11,13,14,15,16,17].”
BackgroundOpen science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540
Self-citationMeta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364
Self-citationHardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Proxies and Databases in Financial Misconduct Research
Karpoff, J. M., & Koester, A. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2811778 · Cited by 329
Meta-research: Why research on research matters
Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005468 · Cited by 314
Self-citationThe Challenge of Applying and Undertaking Research in Female Sport
Emmonds, S., Heyward, O., & Jones, B. (2019) · Sports Medicine - Open · doi:10.1186/s40798-019-0224-x · Cited by 273
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269
Self-citationNaudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227
Self-citationPredictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
Self-citationHardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. A. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620979806 · Cited by 187
Self-citationWang, S. V., Schneeweiss, S., Berger, M. L., Brown, J., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O., Mullins, C. D., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4295 · Cited by 182
Kourou, K., Exarchos, K. P., Papaloukas, C., Sakaloglou, P., Exarchos, T., & Fotiadis, D. I. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.006 · Cited by 165
Replicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 190
Wallach, J. D., Sullivan, P. G., Trepanowski, J. F., Sainani, K. L., Steyerberg, E. W., & Ioannidis, J. P. A. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.9125 · Cited by 148
“The presence of a statistical effect in one subgroup but not the other does not constitute evidence of a subgroup effect, as many authors mistakenly believe; rather, the appropriate statistical approach for establishing a subgroup test is a formal test of interaction.(12) Second, trials often perform numerous subgroup…”
BackgroundSelf-citationReplication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Design principles for creating digital transparency in government
Matheus, R., Janssen, M., & Janowski, T. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101550 · Cited by 267
Assessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139
Self-citationReproducible Research and GIScience: An Evaluation Using GIScience Conference Papers
Ostermann, F. O., Nüst, D., Granell, C., Hofer, B., & Konkol, M. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2021.ii.2 · Cited by 135
What incentives increase data sharing in health and medical research? A systematic review
Rowhani-Farid, A., Allen, M. J., & Barnett, A. G. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0028-9 · Cited by 146
Sustainable computational science: the ReScience initiative
Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F., Brown, C. T., de Buyl, P., Çağlayan, O., Davison, A. P., et al. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.142 · Cited by 141
Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
Semmelrock, H., Ross‐Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2025) · AI Magazine · doi:10.1002/aaai.70002 · Cited by 118
Beyond genomics: understanding exposotypes through metabolomics
Rattray, N. J. W., Deziel, N. C., Wallach, J. D., Khan, S. A., Vasiliou, V., Ioannidis, J. P. A., & Johnson, C. H. (2018) · Human Genomics · doi:10.1186/s40246-018-0134-x · Cited by 131
Self-citationThe Problem with “Magnitude-based Inference”
SAINANI, K. L. (2018) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000001645 · Cited by 117
Reproducible research practices are underused in systematic reviews of biomedical interventions
Page, M. J., Altman, D. G., Shamseer, L., McKenzie, J. E., Ahmadzai, N., Wolfe, D., Yazdi, F., Catalá-López, F., Tricco, A. C., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.017 · Cited by 115
A public database of macromolecular diffraction experiments
Grabowski, M., Langner, K. M., Cymborowski, M., Porebski, P. J., Sroka, P., Zheng, H., Cooper, D. R., Zimmerman, M. D., Elsliger, M.-A., Burley, S. K., et al. (2016) · Acta Crystallographica Section D Structural Biology · doi:10.1107/s2059798316014716 · Cited by 110
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3018 · Cited by 126
Büttner, F., Howell, D. R., Ardern, C. L., Doherty, C., Blake, C. M., Ryan, J. M., Catena, R. D., Chou, L., Fino, P. C., Rochefort, C., et al. (2019) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-100164 · Cited by 118
Who funded the research behind the Oxford–AstraZeneca COVID-19 vaccine?
Cross, S., Rho, Y., Reddy, H., Pepperrell, T., Rodgers, F., Osborne, R., Eni‐Olotu, A., Banerjee, R., Wimmer, S., & Keestra, S. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-007321 · Cited by 112
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Self-citationJhangiani, R. S., & Biswas‐Diener, R. (2017) · Ubiquity Press eBooks · doi:10.5334/bbc · Cited by 185
Burke, L. M., Sharma, A. P., Heikura, I. A., Forbes, S. F., Holloway, M., McKay, A. K. A., Bone, J. L., Leckey, J. J., Welvaert, M., & Ross, M. L. (2020) · PLOS ONE · doi:10.1371/journal.pone.0234027 · Cited by 96
“Inadequate incentive and opportunity to replicate the results of small-scale research projects have contributed to a crisis of confidence in the reproducibility and transparency of many studies in the social and biomedical science literature [1, 2].”
BackgroundRe-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions
Benureau, F., & Rougier, N. P. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00069 · Cited by 139
EULAR points to consider for the use of big data in rheumatic and musculoskeletal diseases
Gossec, L., Kedra, J., Servy, H., Pandit, A., Stones, S. R., Berenbaum, F., Finckh, A., Baraliakos, X., Stamm, T., Gómez-Cabrero, D., et al. (2019) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2019-215694 · Cited by 97
Did awarding badges increase data sharing inBMJ Open? A randomized controlled trial
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020) · Royal Society Open Science · doi:10.1098/rsos.191818 · Cited by 101
Wallach, J. D., Sullivan, P. G., Trepanowski, J. F., Steyerberg, E. W., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i5826 · Cited by 89
“Previous research suggests that replication studies are rare, 21 and it is common for clinical topics of interest to have evidence that comes from only a single trial.”
BackgroundSelf-citationReproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14325 · Cited by 88
Hamilton, D. G., Hong, K., Fraser, H., Rowhani-Farid, A., Fidler, F., & Page, M. J. (2023) · BMJ · doi:10.1136/bmj-2023-075767 · Cited by 86
Varangot‐Reille, C., Suso‐Martí, L., Romero-Palau, M., Suárez-Pastor, P., & Cuenca‐Martínez, F. (2021) · Journal of Pain · doi:10.1016/j.jpain.2021.12.003 · Cited by 109
Has open data arrived at the British Medical Journal (BMJ) ? An observational study
Rowhani-Farid, A., & Barnett, A. G. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011784 · Cited by 84
Investigating resting-state functional connectivity in the cervical spinal cord at 3 T
Eippert, F., Kong, Y., Winkler, A. M., Andersson, J., Finsterbusch, J., Büchel, C., Brooks, J. C. W., & Tracey, I. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.072 · Cited by 84
Gopal, A. D., Wallach, J. D., Aminawung, J. A., Gonsalves, G., Dal‐Ré, R., Miller, J. E., & Ross, J. S. (2018) · Trials · doi:10.1186/s13063-018-2825-y · Cited by 89
Self-citationHardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78
Self-citationMacleod, M., Collings, A. M., Graf, C., Kiermer, V., Mellor, D., Swaminathan, S., Sweet, D., & Vinson, V. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103238118 · Cited by 77
Epistemonikos: a comprehensive database of systematic reviews for health decision-making
on behalf of Epistemonikos project, Rada, G., Pérez, D., Araya‐Quintanilla, F., Ávila, C., Bravo‐Soto, G., Bravo-Jeria, R., Cánepa, A., Capurro, D., Castro-Gutiérrez, V., et al. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01157-x · Cited by 87
Hooijmans, C. R., Hlavica, M., Schuler, F. A. F., Good, N., Good, A., Baumgartner, L., Galeno, G., Schneider, M. P., Jung, T., de Vries, R., et al. (2019) · Scientific Reports · doi:10.1038/s41598-018-35734-4 · Cited by 72
Understanding the care.data conundrum: New information flows for economic growth
Vezyridis, P., & Timmons, S. (2017) · Big Data & Society · doi:10.1177/2053951716688490 · Cited by 79
Kidwell, M. C., Lazarević, L. B., Baranski, E. N., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Słowik, A., Sonnleitner, C., Holden, C. L., et al. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/khbvy · Cited by 88
López‐Nicolás, R., López-López, J. A., Rubio‐Aparicio, M., & Sánchez‐Meca, J. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01644-z · Cited by 80
A critical reexamination of doing arithmetic nonconsciously
Moors, P., & Hesselmann, G. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1292-x · Cited by 66
Hakoum, M. B., Anouti, S., Al‐Gibbawi, M., Abou-Jaoude, E. A., Hasbani, D. J., Lopes, L. C., Agarwal, A., Guyatt, G. H., & Akl, E. A. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011997 · Cited by 61
Loladze, I., Nolan, J. M., Ziska, L. H., & Knobbe, A. R. (2019) · Molecular Nutrition & Food Research · doi:10.1002/mnfr.201801047 · Cited by 62
Wallach, J. D., Serghiou, S., Chu, L., Egilman, A. C., Vasiliou, V., Ross, J. S., & Ioannidis, J. P. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0914-6 · Cited by 57
Self-citationA Critical Analysis of Databases used in Financial Misconduct ResearchPreprint
Karpoff, J. M., Koester, A., Lee, D. S., & Martin, G. S. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2112569 · Cited by 64
The influence of the team in conducting a systematic review
Uttley, L., & Montgomery, P. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0548-x · Cited by 54
“Indeed, many systematic reviews are receiving increasing criticism for failing to live up to their reputation as high-quality, well-conducted pieces of research [8, 9].”
BackgroundSchulz, J. B., Cookson, M. R., & Hausmann, L. (2016) · Journal of Neurochemistry · doi:10.1111/jnc.13844 · Cited by 52
Smith, M. M., Clarke, E., & Little, C. B. (2016) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2016.10.016 · Cited by 55
Hardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M. C., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/39cfb · Cited by 64
Towards a more reproducible ecology
Borregaard, M. K., & Hart, E. (2016) · Ecography · doi:10.1111/ecog.02493 · Cited by 51
Khouri, C., Revol, B., Lepelley, M., Mouffak, A., Bernardeau, C., Salvo, F., Pariente, A., Roustit, M., & Cracowski, J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.07.014 · Cited by 51
Best practice data life cycle approaches for the life sciences
Griffin, P. C., Khadake, J. R., Lemay, K. S., Lewis, S., Orchard, S. E., Pask, A. J., Pope, B. J., Roessner, U., Russell, K., Seemann, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12344.2 · Cited by 55
Kosmidou, C., Efstathiou, N. E., Hoang, M. V., Notomi, S., Konstantinou, E. K., Hirano, M., Takahashi, K., Maidana, D. E., Tsoka, P., Young, L., et al. (2018) · Scientific Reports · doi:10.1038/s41598-017-17634-1 · Cited by 47
Solmi, M., Correll, C. U., Carvalho, A. F., & Ioannidis, J. P. A. (2018) · Epidemiology and Psychiatric Sciences · doi:10.1017/s204579601800032x · Cited by 44
Self-citationReproducible research practices and transparency across linguistics
Bochyńska, A., Keeble, L., Halfacre, C. A. E., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011239 · Cited by 54
Data Privacy and Ethical Considerations in Database Management
Pina, E., Ramos, J., Jorge, H., Váz, P., da Silva, J. P. L. R., Wanzeller, C., Abbasi, M., & Martins, P. (2024) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp4030024 · Cited by 62
Antunes, B. A., & Hill, D. R. C. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100655 · Cited by 78
Sekula, P., Mallett, S., Altman, D. G., & Sauerbrei, W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0178531 · Cited by 40
Catalá-López, F., Caulley, L., Ridao-López, M., Hutton, B., Husereau, D., Drummond, M., Alonso‐Arroyo, A., Pardo-Fernández, M., Bernal‐Delgado, E., Meneu, R., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034463 · Cited by 41
Ethics of Outbreaks Position Statement. Part 1: Therapies, Treatment Limitations, and Duty to Treat
Papadimos, T. J., Marcolini, E. G., Hadian, M., Hardart, G. E., Ward, N., Levy, M. M., Stawicki, S. P., & Davidson, J. E. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003416 · Cited by 36
Scientific rigor and credibility in the nutrition research landscape
Kroeger, C. M., Garza, C., Lynch, C. J., Myers, E. F., Rowe, S. B., Schneeman, B. O., Sharma, A. M., & Allison, D. B. (2018) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqx067 · Cited by 45
Embedding data provenance into the Learning Health System to facilitate reproducible research
Ćurčin, V. (2016) · Learning Health Systems · doi:10.1002/lrh2.10019 · Cited by 44
EnzymeML: seamless data flow and modeling of enzymatic data
Lauterbach, S., Dienhart, H., Range, J., Malzacher, S., Spöring, J., Rother, D., Pinto, M. F., Martins, P. M., Lagerman, C. E., Bommarius, A. S., et al. (2023) · Nature Methods · doi:10.1038/s41592-022-01763-1 · Cited by 52
A manifesto for reproducible science.
Munafo, M. R., Nosek, B. A., Bishop, D., Button, K. S., Chambers, C., du Sert, N. P., Simonsohn, U., Wagenmakers, E., Ware, J. J., & Ioannidis, J. P. A. (2024) · American Psychological Association eBooks · doi:10.1037/0000409-041 · Cited by 42
Self-citationToward assessing clinical trial publications for reporting transparency
Kilicoglu, H., Rosemblat, G., Hoang, L., Wadhwa, S., Peng, Z., Malički, M., Schneider, J., & ter Riet, G. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103717 · Cited by 34
Data sharing in orthodontic research
Papageorgiou, S. N., & Cobourne, M. T. (2018) · Journal of Orthodontics · doi:10.1080/14653125.2018.1440792 · Cited by 34
All science should inform policy and regulation
Ioannidis, J. P. A. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002576 · Cited by 33
Self-citationDenaxas, S., Direk, K., González-Izquierdo, A., Pikoula, M., Cakiroglu, S. A., Moore, J. H., Hemingway, H., & Smeeth, L. L. (2017) · BioData Mining · doi:10.1186/s13040-017-0151-7 · Cited by 35
The Replication Crisis in Epidemiology: Snowball, Snow Job, or Winter Solstice?
Lash, T. L., Collin, L. J., & Dyke, M. E. V. (2018) · Current Epidemiology Reports · doi:10.1007/s40471-018-0148-x · Cited by 39
Granan, L., Rème, S. E., Jacobsen, H. B., Stubhaug, A., & Ljosaa, T. M. (2019) · Scandinavian Journal of Pain · doi:10.1515/sjpain-2017-0160 · Cited by 39
Independent discussion sections for improving inferential reproducibility in published research
Avidan, M. S., Ioannidis, J. P. A., & Mashour, G. A. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2018.12.010 · Cited by 34
Self-citationImproving Disclosure of Financial Conflicts of Interest for Research on Psychosocial Interventions
Cristea, I. A., & Ioannidis, J. P. A. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.0382 · Cited by 31
Self-citationTransparency and Public Involvement in Animal Research
Pound, P., & Blaug, R. (2016) · Alternatives to Laboratory Animals · doi:10.1177/026119291604400210 · Cited by 40
Qu, Y., Wang, P., Liu, B., Song, C., Wang, D., Yang, H., Zhang, Z., Chen, P., Kang, X., Du, K., et al. (2021) · Brain Disorders · doi:10.1016/j.dscb.2021.100005 · Cited by 29
Dumas-Mallet, E., Button, K. S., Boraud, T., Munafo, M. R., & Gonon, F. (2016) · PLoS ONE · doi:10.1371/journal.pone.0158064 · Cited by 31
A New Account of Replication in the Experimental Life Sciences
Guttinger, S. (2019) · Philosophy of Science · doi:10.1086/703555 · Cited by 27
Badges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 53
Sun, Q., Welsh, K. J., Bruns, D. E., Sacks, D. B., & Zhao, Z. (2019) · Clinical Chemistry · doi:10.1373/clinchem.2019.309575 · Cited by 29
Open spatial sciences: an introduction
Páez, A. (2021) · Journal of Geographical Systems · doi:10.1007/s10109-021-00364-4 · Cited by 25
EPIGEN-Brazil Initiative resources: a Latin American imputation panel and the Scientific Workflow
Magalhães, W. C. S., Araujo, N. M., Leal, T. P., de Araújo, G. S., Viriato, P. J. S., Kehdy, F. S. G., Costa, G. N. D. O., Barreto, M. L., Horta, B. L., Lima‐Costa, M. F., et al. (2018) · Genome Research · doi:10.1101/gr.225458.117 · Cited by 28
Reproducible and transparent research practices in published neurology research
Rauh, S. L., Torgerson, T., Johnson, A., Pollard, J., Tritz, D., & Vassar, M. B. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-0091-5 · Cited by 27
Páez, A. (2021) · Geographical Analysis · doi:10.1111/gean.12307 · Cited by 27
Yeung, A. W. K., Robertson, M., Uecker, A. M., Fox, P. T., & Eickhoff, S. B. (2022) · Human Brain Mapping · doi:10.1002/hbm.26177 · Cited by 26
Koçak, B., Yardımcı, A. H., Yüzkan, S., Keles, A., Altun, O., Bulut, E., Bayrak, O. N., & Okumus, A. A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.11.030 · Cited by 31
Data Management Practices in Academic Library Learning Analytics: A Critical Review
Briney, K. A. (2019) · Journal of Librarianship and Scholarly Communication · doi:10.7710/2162-3309.2268 · Cited by 36
Metadata
- Authors
- Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A.
- Affiliations
- Emory University (US)
- Stanford Medicine (US)
- Centers for Disease Control and Prevention (US)
- National Institutes of Health (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-01-04
- Volume
- 14
- Issue
- 1
- Pages
- e1002333
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4699702
- OpenAlex
- W2228927609
- Cited by
- 310
- References
- 41
- Citations by year
- 2026: 12 · 2025: 16 · 2024: 19 · 2023: 26 · 2022: 19 · 2021: 38 · 2020: 33 · 2019: 40 · 2018: 42 · 2017: 31 · 2016: 33
- FWCI
- 92.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Biology (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- reproducibility, research transparency, biomedical literature, data availability, raw data sharing, replication study, systematic reviews, funding disclosure, conflicts of interest
- MeSH terms
- Periodicals as Topic, Reproducibility of Results, Conflict of Interest, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

