The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Transparency and reproducibility in artificial intelligence
Open accessAbstract
Open Access articles citing this article. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications Kenniy Olorunnimbe & Herna Viktor Artificial Intelligence Review Open Access 30 June 2022 Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening Jenny Yang , Andrew A. S. Soltan & David A. Clifton npj Digital Medicine Open Access 07 June 2022 Bringing machine learning to research on intellectual and developmental disabilities: taking inspiration from neurological diseases Chirag Gupta , Pramod Chandrashekar … Daifeng Wang Journal of Neurodevelopmental Disorders Open Access 02 May 2022
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc. 39,40.”
From Artificial intelligence in pancreatic cancer (2022) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“highlighting the risk of using corporate models in a sensitive area such as medicine, where patient privacy, transparency and reliability are essential74,76”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication -including associations of reproducibility and rigour [21, 23]- without actually doing the work.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [23]
In the citing paper
“One of the most effective methods of mitigating AI biases is to ensure the use of diverse and representative datasets during AI development and training [67, 87].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [87]
In the citing paper
“A critical component of AI evaluation and development more generally is ensuring transparency and reproducibility, allowing independent research groups to review and build on prior scientific progress (196).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [196]
In the citing paper
“In addition, the lack of transparency and reproducibility of AI computational algorithm methods and code in many published studies threatens scientific progress and discovery, which require that independent scientists are able to scrutinize and reproduce results98.”
From Smart wearable devices in cardiovascular care: Where we are and how to move forward (2021) · cites this paper as [98]
In the citing paper
“Because of the particular significance of addressing the data-related concerns, different countries have developed policies and laws [160].”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [160]
In the citing paper
“Unfortunately, though, the majority of existing AI/ML models remain in specialized journals, with poorly maintained code and variable rigor in validation.283”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [283]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
(2015) · neural information processing systems · Cited by 912
A curated mammography data set for use in computer-aided detection and diagnosis research
Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017) · Scientific Data · doi:10.1038/sdata.2017.177 · Cited by 859
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 573
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · cited 1×
“Even with extensive description, reproducing complex computational pipelines based purely on text is a subjective and challenging task 9 .”
Background
Research works printed but not linked: 1
- Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).
Cited by
Sorted by citations
Items 101–200 of 541
Davidson, E. M., Poon, M. T. C., Casey, A., Grivas, A., Duma, D., Dong, H., Suárez-Paniagua, V., Grover, C., Tobin, R., Whalley, H. C., et al. (2021) · BMC Medical Imaging · doi:10.1186/s12880-021-00671-8 · Cited by 37
Distributed Cooperative Coevolution of Data Publishing Privacy and Transparency
Ge, Y.-F., Bertino, E., Wang, H., Cao, J., & Zhang, Y. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3613962 · Cited by 40
“Recently, the quest for data transparency regarding AI has been emphasized [24].”
BackgroundAdvances in AI‐based cancer cytopathology
Yang, Y., Guan, S., Ou, Z., Li, W., Yan, L., & Situ, B. (2023) · Interdisciplinary medicine · doi:10.1002/inmd.20230013 · Cited by 37
Machine Learning to Advance Human Genome-Wide Association Studies
Sigala, R. E., Lagou, V., Shmeliov, A., Atito, S., Kouchaki, S., Awais, M., Prokopenko, I., Mahdi, A., & Demirkan, A. (2023) · Genes · doi:10.3390/genes15010034 · Cited by 41
Best practices for data management and sharing in experimental biomedical research
Cunha‐Oliveira, T., Ioannidis, J. P. A., & Oliveira, P. J. (2024) · Physiological Reviews · doi:10.1152/physrev.00043.2023 · Cited by 47
Self-citationExploiting Blockchain to Make AI Trustworthy: A Software Development Lifecycle View
Zhang, P., Ding, S., & Zhao, Q. (2023) · ACM Computing Surveys · doi:10.1145/3614424 · Cited by 46
“The second one includes fairness, explainability, privacy, and auditability.”
BackgroundNg, M. Y., Youssef, A., Miner, A. S., Sarellano, D., Long, J., Larson, D. B., Hernandez‐Boussard, T., & Langlotz, C. P. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.45892 · Cited by 40
Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., et al. (2023) · Radiographics · doi:10.1148/rg.230180 · Cited by 35
“…the analysis to enable scientific understanding of how the analysis was performed [15]) and limited reproducibility (the ability to replicate the analysis given the same input data [16]) of AI imaging analysis workflows is broadly recognized as a major obstacle to clinical translation (15).”
Self-citationMAIP: a web service for predicting blood‐stage malaria inhibitors
Bosc, N., Félix, E., Arcila, R., Méndez, D., Saunders, M. R., Green, D. V. S., Ochoada, J., Shelat, A. A., Martin, É., Iyer, P., et al. (2021) · Journal of Cheminformatics · doi:10.1186/s13321-021-00487-2 · Cited by 43
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 48
“ML faces analogous issues to the lack of open data and code in social psychology [68, 99, 107, 182, 197].”
BackgroundCovidCTNet: An Open-Source Deep Learning Approach to Identify Covid-19 Using CT ImagePreprint
Javaheri, T., Homayounfar, M., Amoozgar, Z., Reiazi, R., Homayounieh, F., Abbas, E., Laali, A., Radmard, A. R., Gharib, M. H., Mousavi, S. A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03059 · Cited by 36
Self-citationTowards intellectual freedom in an AI Ethics Global Community
Ebell, C., Baeza-Yates, R., Benjamins, R., Cai, H., Coeckelbergh, M., Duarte, T., Hickok, M., Jacquet, A., Kim, A., Krijger, J., et al. (2021) · AI and Ethics · doi:10.1007/s43681-021-00052-5 · Cited by 34
“Problems of systemic racism and other culturally deeply encoded discriminatory mechanisms multiply the severity of such conflicts of interest insignificant, perhaps non-linear and even counter-intuitive ways [22, 24, 36, 38, 39, 44].”
BackgroundContinental generalization of a human-in-the-loop AI system for clinical seizure recognition
Yang, Y., Truong, N. D., Maher, C., Nikpour, A., & Kavehei, O. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118083 · Cited by 39
Wagner, S. J., Matek, C., Boushehri, S. S., Boxberg, M., Lamm, L., Sadafi, A., Waibel, D., Marr, C., & Peng, T. (2023) · Modern Pathology · doi:10.1016/j.modpat.2023.100350 · Cited by 34
Valtonen, L., Mäkinen, S., & Kirjavainen, J. (2022) · Organizational Research Methods · doi:10.1177/10944281221124947 · Cited by 31
“Vague methodological descriptions undermine the reproducibility of UML research, thereby limiting the potential to ascertain its validity and reproducibility and hindering other researchers ’ possibilities of building on the results (Haibe-Kains et al., 2020; Zhang & Shaw, 2012).”
BackgroundMethodAcademics’ Weak(ening) Resistance to Generative AI: The Cause and Cost of Prestige?
Watermeyer, R., Lanclos, D., Phipps, L., Shapiro, H., Guizzo, D., & Knight, C. (2024) · Postdigital Science and Education · doi:10.1007/s42438-024-00524-x · Cited by 49
“…inter alia to concerns of privacy, ethics, data bias, and poisoning (Arora et al. 2023; Mikalef et al. 2022; Verde et al. 2021); over-reliance, misuse, and responsible use (Chan and Colloton 2024); scientific misinformation (Truhn et al. 2023); and irreproducible research (Haibe-Kains et al. 2020).”
Pathak, A., Theagarajan, R., Rizqi, M. M., Nugraha, A. S., Boruah, T., Pratiksha, Kumar, H., Naik, B., Yadav, S., Jha, A. K., et al. (2025) · Discover Molecules · doi:10.1007/s44345-025-00037-5 · Cited by 34
“Ensuring reproducibility requires a concerted effort towards open science practices, including sharing code, models, and fully specified training workflows, as well as the adoption of community-agreed-upon benchmarks to provide a reliable measure of progress and utility [230].”
Reply to: Transparency and reproducibility in artificial intelligence
McKinney, S. M., Karthikesalingam, A., Tse, D., Kelly, C. J., Liu, Y., Corrado, G. S., & Shetty, S. (2020) · Nature · doi:10.1038/s41586-020-2767-x · Cited by 32
TPCx-AI - An Industry Standard Benchmark for Artificial Intelligence and Machine Learning Systems
Brücke, C., Härtling, P., Palacios, R. E., Patel, H., & Rabl, T. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611540.3611554 · Cited by 32
Robustness and reproducibility for AI learning in biomedical sciences: RENOIR
Barberis, A., Aerts, H. J. W. L., & Buffa, F. M. (2024) · Scientific Reports · doi:10.1038/s41598-024-51381-4 · Cited by 33
Self-citationFactors in the adoption of open government initiatives in Spanish local governments
Muñoz, L. A., Bolívar, M. P. R., & Arellano, C. L. V. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101743 · Cited by 29
Kim, Y. K., Seo, W.-D., Lee, S. J., Koo, J. H., Kim, G. C., Song, H. S., & Lee, M. (2024) · Journal of Medical Internet Research · doi:10.2196/62890 · Cited by 31
Galaxy-ML: An accessible, reproducible, and scalable machine learning toolkit for biomedicine
Gu, Q., Kumar, A., Bray, S. A., Creason, A., Khanteymoori, A., Jalili, V., Grüning, B. A., & Goecks, J. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009014 · Cited by 28
“This is critical, as reproducibility has become critical in machine learning research [13,14].”
Backgroundhistolab: A Python library for reproducible Digital Pathology preprocessing with automated testing
Marcolini, A., Bussola, N., Arbitrio, E., Amgad, M. T., Jurman, G., & Furlanello, C. (2022) · SoftwareX · doi:10.1016/j.softx.2022.101237 · Cited by 28
Self-citationAI for development: implications for theory and practice
Bjola, C. (2021) · Oxford Development Studies · doi:10.1080/13600818.2021.1960960 · Cited by 29
ViLMedic: a framework for research at the intersection of vision and language in medical AI
Delbrouck, J.-B., Saab, K. K., Varma, M., Eyuboglu, S., Chambon, P., Dunnmon, J., Chaves, J. M. Z., Chaudhari, A. S., & Langlotz, C. P. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.3 · Cited by 30
Thy‐Wise : An interpretable machine learning model for the evaluation of thyroid nodules
Jin, Z., Pei, S., Ouyang, L., Zhang, L., Mo, X., Chen, Q., You, J., Chen, L., Zhang, B., & Zhang, S. (2022) · International Journal of Cancer · doi:10.1002/ijc.34248 · Cited by 27
“38 The prediction process will need to be transparent to ensure medical professionals embrace this new technology.”
BackgroundMethodKempers, S., van Lierop, P., Hsu, T.-M. H., Moin, D. A., Bergé, S. J., Ghaeminia, H., Xi, T., & Vinayahalingam, S. (2023) · Journal of Dentistry · doi:10.1016/j.jdent.2023.104519 · Cited by 27
Deep Learning‐Based Multiparametric MRI Model for Preoperative T‐Stage in Rectal Cancer
Wei, Y., Wang, H., Chen, Z., Zhu, Y., Li, Y., Lu, B., Pan, K., Wen, C., Cao, G., He, Y., et al. (2023) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.28856 · Cited by 26
“14 Previous research has shown that deep learning performs splendidly in a variety of oncology scenarios, including evaluating tumor biological behavior, 15,16 predicting treatment response, 17,18 and recognizing unfavorable treatment outcomes.”
BackgroundKang, D., Kang, T., & Jang, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103477 · Cited by 28
Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology
Sauter, D., Lodde, G., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M. (2022) · Sensors · doi:10.3390/s22145346 · Cited by 26
“We apply two XAI methods to histopathology: ACE and Guided Grad-CAM as examples for explanations on model level and image level, respectively.”
MethodData Provenance in Biomedical Research: Scoping Review
Johns, M., Meurers, T., Wirth, F. N., Haber, A. C., Müller, A., Halilovic, M., Balzer, F., & Praßer, F. (2023) · Journal of Medical Internet Research · doi:10.2196/42289 · Cited by 29
“…as both types of data are rather large and complex in nature, and processing operations, for example, bioinformatics pipelines or artificial intelligence–based image analyses, are known to sometimes be difficult to reproduce [69,70] (refer to the Principal Findings section for further discussion).”
Wilczok, D. (2025) · Aging · doi:10.18632/aging.206190 · Cited by 32
Replication studies in the clinical decision support literature–frequency, fidelity, and impact
Coiera, E. W., & Tong, H. L. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab049 · Cited by 27
O’Shea, R. J., Sharkey, A. R., Cook, G., & Goh, V. (2021) · European Radiology · doi:10.1007/s00330-021-07881-2 · Cited by 28
“Many commentators have advocated for transparency in clinical AI research [19, 38, 40, 42, 43, 53, 54].”
BackgroundKoçak, B., Yardımcı, A. H., Yüzkan, S., Keles, A., Altun, O., Bulut, E., Bayrak, O. N., & Okumus, A. A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.11.030 · Cited by 27
Olsen, H. P., Hildebrandt, T., Wiesener, C., Larsen, M. S., & Flügge, A. W. A. (2023) · Digital Government Research and Practice · doi:10.1145/3632753 · Cited by 44
“Other researchers have embarked on the challenging task of devising methods for transparency by design [ 34 ], transparency and reproducibility [ 20 ], and trustworthy AI using formal verification of the correctness of AI algorithms [ 23 ].”
MethodMolecular MRI-Based Monitoring of Cancer Immunotherapy Treatment Response
Vladimirov, N., & Perlman, O. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24043151 · Cited by 25
The Adoption of Artificial Intelligence in Bureaucratic Decision-making: A Weberian Perspective
Presuel, R. C., & Sierra, J. M. M. (2023) · Digital Government Research and Practice · doi:10.1145/3609861 · Cited by 40
“…in the adoption of technology, to meeting standards of reproductivity and validation, to being able to understand how AI systems function [116] While it can be argued that the embedding of technology into public administration decision-making has led to increases in oversight and…”
BackgroundArtificial Intelligence in the Differential Diagnosis of Cardiomyopathy Phenotypes
Cau, R., Pisu, F., Suri, J. S., Montisci, R., Gatti, M., Mannelli, L., Gong, X., & Saba, L. (2024) · Diagnostics · doi:10.3390/diagnostics14020156 · Cited by 25
Baldini, B., Papasratorn, D., Fagundes, F. B., Fontenele, R. C., & Jacobs, R. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105546 · Cited by 23
DeepISLES: a clinically validated ischemic stroke segmentation model from the ISLES'22 challenge
de la Rosa, E., Reyes, M., Liew, S., Hutton, A., Wiest, R., Kaesmacher, J., Hanning, U. M. P., Hakim, A., Zubal, R., Valenzuela, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62373-x · Cited by 43
“Validating DeepISLES in external data .”
Rise of Generative Artificial Intelligence in Science
Ding, L., Lawson, C., & Shapira, P. (2024) · Scientometrics · doi:10.1007/s11192-025-05413-z · Cited by 41
“…to revolutionize science and technology, sparking widespread discussions about its implications, including for ethics (OECD, 2023), reproducibility (Haibe-Kains et al., 2020), employment (Septiandri et al., 2024), productivity (Filippucci et al., 2024), and creativity (Mukherjee & Chang, 2024).”
Ethical Considerations and Fairness in the Use of Artificial Intelligence for Neuroradiology
Filippi, C. G., Stein, J. M., Wang, Z., Bakas, S., Liu, Y., Chang, P., Lui, Y. W., Hess, C. P., Barboriak, D. P., Flanders, A. E., et al. (2023) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7963 · Cited by 32
Automatic Classification of Cardiac Arrhythmias Using Deep Learning Techniques: A Systematic Review
Vásquez-Iturralde, F., Flores-Calero, M. J., Grijalva, F., & Rosales, A. (2024) · IEEE Access · doi:10.1109/access.2024.3408282 · Cited by 23
“The lack of detailed methods and algorithm codes undermines its scientific value [128].”
BackgroundDistribution-invariant differential privacy
Bi, X., & Shen, X. (2022) · Journal of Econometrics · doi:10.1016/j.jeconom.2022.05.004 · Cited by 22
Make deep learning algorithms in computational pathology more reproducible and reusable
Wagner, S. J., Matek, C., Boushehri, S. S., Boxberg, M., Lamm, L., Sadafi, A., Waibel, D., Marr, C., & Peng, T. (2022) · Nature Medicine · doi:10.1038/s41591-022-01905-0 · Cited by 21
Economic recommender systems – a systematic review
Biasio, A. D., Navarin, N., & Jannach, D. (2023) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2023.101352 · Cited by 32
Transparency, bias, and reproducibility across science: a meta-research view
Ioannidis, J. P. A. (2024) · Journal of Clinical Investigation · doi:10.1172/jci181923 · Cited by 24
“Transparency, nevertheless, is feasible even in the most complex AI methods (48).”
Self-citationMehrnia, S. S., Safahi, Z., Mousavi, A., Panahandeh, F., Farmani, A., Yuan, R., Rahmim, A., & Salmanpour, M. R. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01458-x · Cited by 20
A white paper on good research practices in benchmarking: The case of cluster analysis
Mechelen, I. V., Boulesteix, A., Dangl, R., Dean, N., Hennig, C. M., Leisch, F., Steinley, D., & Warrens, M. J. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1511 · Cited by 56
“…to their foundations by powerful calls for scientific integrity, for buttressing up claims in evidence-based ways, for setting up studies in reproducible and replicable manners, and for adhering in all possible respects to good research practices (see, e.g., Haibe-Kains et al., 2020; Hutson, 2018).”
BackgroundEnd-to-end reproducible AI pipelines in radiology using the cloud
Bontempi, D., Nürnberg, L., Pai, S., Krishnaswamy, D., Thiriveedhi, V., Hosny, A., Mak, R. H., Farahani, K., Kikinis, R., Fedorov, A., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51202-2 · Cited by 23
Self-citationFederated Machine Learning Enables Risk Management and Privacy Protection in Water Quality
Wang, Y., Wang, H., Wang, W., Yang, H., Chen, J.-J., Fan, Y.-X., Yin, W.-X., Lv, J.-Q., Luo, X.-Q., Zhou, X. H., et al. (2025) · Environmental Science & Technology · doi:10.1021/acs.est.5c00369 · Cited by 19
Cross-Regional Analysis of Global AI Healthcare Regulation
Ullagaddi, P. (2025) · Journal of Computer and Communications · doi:10.4236/jcc.2025.135005 · Cited by 18
Pre-registration of mathematical models
Ioannidis, J. P. A. (2022) · Mathematical Biosciences · doi:10.1016/j.mbs.2022.108782 · Cited by 20
Self-citationBuvat, I., & Orlhac, F. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.120.261586 · Cited by 22
“The reproducibility crisis affects many fi elds and has been extensively studied and debated ( 6 ), including in the fi eld of radiology ( 7,8 ).”
BackgroundTowards more replicable content analysis for learning analytics
Kitto, K., Manly, C. A., Ferguson, R., & Poquet, O. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576096 · Cited by 18
“For example, in 2015 an attempt to replicate results from psychology found that more than half the field’s significant results failed reproducibility tests [4], and concerns about the dependence of artificial intelligence (AI) upon training datasets that are rarely released to the public [22] are leading to critical d…”
The Evolution and Reliability of Machine Learning Techniques for Oncology
Owida, H. A., Moh’d, B. A.-H., Turab, N. M., Al-Nabulsi, J., & Abuowaida, S. F. A. (2023) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v19i08.39433 · Cited by 24
EEG Datasets for Healthcare: A Scoping Review
da Silva, C. P., Tedesco, S. A., & O’Flynn, B. (2024) · IEEE Access · doi:10.1109/access.2024.3376254 · Cited by 19
“Even if one has access to the original data, it is not always possible to reproduce ML results [26], [27].”
BackgroundIntegrating artificial intelligence in bedside care for covid-19 and future pandemics
Yu, M. A., Tang, A., Brown, K., Bouchakri, R., St-Onge, P., Wu, S., Reeder, J. C., Mullie, L., & Chassé, M. (2021) · BMJ · doi:10.1136/bmj-2021-068197 · Cited by 17
Bussola, N., Papa, B., Melaiu, O., Castellano, A., Fruci, D., & Jurman, G. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22168804 · Cited by 16
“…robustness and reproducibility to the modeling, a preliminary training/test split with ratio 3 / 4 − 1 / 4 was operated and on the training set a 4 × 5 − cross validation resampling strategy was employed, following the guidelines recommended by the US-FDS in their MAQC/SEQC initiatives [129,130].”
MethodReynolds, R. J., Scott, R. T., Turner, R. T., Iwaniec, U. T., Bouxsein, M., Sanders, L., & Antonsen, E. (2022) · Biomedicines · doi:10.3390/biomedicines10092187 · Cited by 18
“This also included ensuring ALSDA datasets were maximally open access [15], FAIR (findable, accessible, interoperable, reusable) [16], and data files were curated to be machine-readable (e.g., “tidy” data format) [17], transparent, reproducible, and maximally “AI-ready” [13,18,19]).”
Steurer, B., Vanhaelen, Q., & Zhavoronkov, A. A. (2024) · The Journals of Gerontology Series A · doi:10.1093/gerona/glae006 · Cited by 16
Turrisi, R., Verri, A., & Barla, A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1360095 · Cited by 28
Is artificial intelligence prepared for the 24-h shifts in the ICU?
Gonzalez, F. A., Santonocito, C., Lamas, T., Costa, P. H. P., Vieira, S. M., Ferreira, H. A., & Sanfilippo, F. (2024) · Anaesthesia Critical Care & Pain Medicine · doi:10.1016/j.accpm.2024.101431 · Cited by 22
Artificial Intelligence in Ischemic Heart Disease Prevention
Parsa, S., Shah, P., Doijad, R., & Rodríguez, F. (2025) · Current Cardiology Reports · doi:10.1007/s11886-025-02203-0 · Cited by 18
Artificial intelligence in functional imaging of the lung
Estépar, R. S. J. (2021) · British Journal of Radiology · doi:10.1259/bjr.20210527 · Cited by 20
Machine learning in vadose zone hydrology: A flashback
Ghanbarian, B., & Pachepsky, Y. A. (2022) · Vadose Zone Journal · doi:10.1002/vzj2.20212 · Cited by 17
Evaluating dental AI research papers: Key considerations for editors and reviewers
Uribe, S. E., Hamdan, M. H., Valente, N. A., Yamaguchi, S., Umer, F., Tichý, A., Pauwels, R., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105867 · Cited by 16
Nooraie, R. Y., Lyons, P. G., Baumann, A. A., & Saboury, B. (2021) · PET Clinics · doi:10.1016/j.cpet.2021.07.002 · Cited by 17
Toward a taxonomy of trust for probabilistic machine learning
Broderick, T., Gelman, A., Meager, R., Smith, A. L., & Zheng, T. (2023) · Science Advances · doi:10.1126/sciadv.abn3999 · Cited by 19
“Some do not even describe their algorithm in enough detail for the reader to produce equivalent code (Haibe-Kains et al., 2020).”
BackgroundSelf-citationHow Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models?
Shahriari, M., Ramler, R., & Fischer, L. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4040045 · Cited by 16
“However, repeatability issues are frequent in DL [15] and, in consequence, DL is facing a serious reproducibility challenge [35,36] which is gaining more and more attention in the research community.”
BackgroundPerspectives on development of biomedical polymer materials in artificial intelligence age
Xie, S. (2023) · Journal of Biomaterials Applications · doi:10.1177/08853282231151822 · Cited by 16
“56,57 To understand the meaning of direct design, the algorithm complexity must be quantitatively compared with counting.”
BackgroundChristopher, M. A., Gonzalez, R., Huynh, J., Walker, E., Saseendrakumar, B. R., Bowd, C., Belghith, A., Goldbaum, M. H., Fazio, M. A., Girkin, C. A., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11020140 · Cited by 16
Esteban, L. M., Borque‐Fernando, Á., Escorihuela, M. E., Esteban‐Escaño, J., Junquera, J. M. A., Servián, P., & Moróte, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-88297-6 · Cited by 17
Empowering nanophotonic applications via artificial intelligence: pathways, progress, and prospects
Chen, W., Yang, S., Yan, Y., Gao, Y., Zhu, J., & Dong, Z. (2025) · Nanophotonics · doi:10.1515/nanoph-2024-0723 · Cited by 17
The Hallmarks of Predictive Oncology
Singhal, A., Zhao, X., Wall, P., So, E., Calderini, G., Partin, A., Koussa, N. C., Vasanthakumari, P., Narykov, O., Zhu, Y., et al. (2025) · Cancer Discovery · doi:10.1158/2159-8290.cd-24-0760 · Cited by 19
“Nonetheless, there are many published models that do not share data, computer code, or fitted models, calling for more transparency in AI research (119).”
Self-citationDiscovery of predictive biomarkers for cancer therapy through computational approaches
Wang, X., Nguyen, J., Nader, K. M., Miihkinen, M., Wall, P., Singhal, A., Bedard, P. L., Ideker, T., Aittokallio, T. A., & Haibe‐Kains, B. (2026) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-01109-8 · Cited by 24
“Open science is an ongoing effort to make scientific research transparent, reproducible and reusable 181 .”
Self-citationReproducibility in Subsurface Geoscience
Steventon, M., Jackson, C. A.-L., Ireland, M., Hall, M., Munafo, M. R., & Roberts, K. (2022) · Earth Science Systems and Society · doi:10.3389/esss.2022.10051 · Cited by 15
A Systematic Review of Ethics Disclosures in Predictive Mental Health Research
Ajmani, L., Chancellor, S., Mehta, B., Fiesler, C. L., Zimmer, M. T., & Choudhury, M. D. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594082 · Cited by 17
Yan, Y., Huang, X., Jiang, X., Gao, Z., Liu, X., Jin, K., & Ye, J. (2023) · Eye · doi:10.1038/s41433-023-02765-9 · Cited by 15
“Clinicians’ lack of understanding of AI and the inherent inexplicability of DL decrease its acceptability [26, 27].”
Rostami-Tabar, B., Ali, M., Hong, T., Hyndman, R. J., Porter, M. D., & Syntetos, A. A. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.02.010 · Cited by 15
Backdoor Attacks to Deep Neural Network-Based System for COVID-19 Detection from Chest X-ray Images
Matsuo, Y., & Takemoto, K. (2021) · Applied Sciences · doi:10.3390/app11209556 · Cited by 16
“Such open-source projects are encouraging not only for developing high-performance DNN solutions, but also for ensuring transparency and reproducibility in DNN models [21], although only deep learning models (model weights) may be provided [22] as an alternative to sharing patient data with regard to preserving patien…”
Wang, Y., Zhou, H.-B., Luo, X.-Q., Deng, S.-W., Xu, H., Song, Y., Chen, J.-J., Yin, W.-X., Cheng, H.-Y., Wang, A.-J., et al. (2025) · Environmental Science & Technology · doi:10.1021/acs.est.5c06126 · Cited by 14
Bolocan, V.-O., Mitoi, A., Nicu-Canareica, O., Băean, M.-L., Medar, C., & Popa, G.-A. (2025) · Journal of Imaging · doi:10.3390/jimaging11100335 · Cited by 19
“Our analysis reveals systemic methodological issues undermining evidence quality and reproducibility in AI prostate MRI research [79].”
Pepe, A., Egger, J., Codari, M., Willemink, M. J., Gsaxner, C., Li, J., Roth, P. M., Schmalstieg, D., Mistelbauer, G., & Fleischmann, D. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107365 · Cited by 14
The state of artificial intelligence in pediatric urology
Khondker, A., Kwong, J. C. C., Malik, S., Erdman, L., Keefe, D. T., Fernández, N., Tasian, G. E., Wang, H. S., Estrada, C. R., Nelson, C. P., et al. (2022) · Frontiers in Urology · doi:10.3389/fruro.2022.1024662 · Cited by 14
Reproducibility in NLP: What Have We Learned from the Checklist?
Magnusson, I., Smith, N. A., & Dodge, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.809 · Cited by 22
Pouchard, L. C., Reyes, K. G., Alexander, F. J., & Yoon, B.-J. (2023) · Digital Discovery · doi:10.1039/d3dd00094j · Cited by 15
“Published results for ML experiments often privilege accuracy obtained with much tuning, and the publications reporting these results may not necessarily provide the ranges of input conditions that produce the reported accuracy, resulting in irreproducible results [7,8].”
ResultHong, W., Liang, C., Ma, Y., & Zhu, J. (2023) · Systems · doi:10.3390/systems11110551 · Cited by 16
The impact of artificial intelligence on Ukrainian medicine: benefits and challenges for the future
Sofilkanych, N., Vesova, O., Kaminskyy, V., & Kryvosheieva, A. (2023) · Futurity Medicine. · doi:10.57125/fem.2023.12.30.04 · Cited by 21
Basic Artificial Intelligence Techniques
Kalpathy–Cramer, J., Patel, J., Bridge, C. P., & Chang, K. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.06.005 · Cited by 15
Singh, R., Mayank Vatsa, A., & Ratha, N. (2020) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3430984.3431966 · Cited by 14
“In other research directions, explainability [9, 14, 27], transparency [25], data lineage [51], privacy and security [13], and social well being [41] have also been addressed.”
Practical resources for enhancing the reproducibility of mechanistic modeling in systems biology
Blinov, M. L., Gennari, J. H., Karr, J. R., Moraru, I. I., Nickerson, D. P., & Sauro, H. M. (2021) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2021.06.001 · Cited by 11
Artificial Intelligence for Earth System Predictability (AI4ESP) (2021 Workshop Report)
Hickmon, N. L., Varadharajan, C., Hoffman, F. M., Wainwright, H. M., & Collis, S. (2022) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1888810 · Cited by 16
Burton‐Jones, A., Boh, W. F., Oborn, E., & Padmanabhan, B. (2021) · MIS Quarterly · doi:10.25300/misq/2021/452e1 · Cited by 17
Rizzi, S., Saroglia, G., Kalemi, V., Rimoldi, S., & Terova, G. (2025) · Applied Sciences · doi:10.3390/app15179781 · Cited by 14
Uses, opportunities and risks of artificial intelligence in participatory urban planning
Raymond, C. M., Nummi, P., von Wirth, T., Poom, A., Ahdekivi, A., Barthel, S., Delmelle, E., Dunkel, E., Fagerholm, N., Grêt-Regamey, A., et al. (2025) · Discover Cities · doi:10.1007/s44327-025-00137-4 · Cited by 15
“Generative AI leads to more pattern recognition without deep understanding by the user, which may proliferate misinformation and could lead to irreproducible results [49, 50].”
Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI
Gkontra, P., Quaglio, G., Garmendia, A. T., & Lekadir, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_17 · Cited by 17
Historical perspective and future directions: computational science in immuno-oncology
Ricker, C. A., Meli, K., & Allen, E. M. V. (2024) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2023-008306 · Cited by 14
Metadata
- Authors
- Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., Mason, C. E., Jones, W., Dopazo, J., Furlanello, C., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C. S., Broderick, T., Hoffman, M. M., Leek, J. T., Korthauer, K., Huber, W., Brazma, A., Pineau, J., Tibshirani, R., Hastie, T., Ioannidis, J. P. A., Quackenbush, J., & Aerts, H. J. W. L.
- Affiliations
- University of Toronto (CA)
- Brigham and Women's Hospital (US)
- University Health Network (CA)
- National Center for Toxicological Research (US)
- University of Oxford (GB)
- SAS Institute (United States) (US)
- Q2 Solutions (United States) (US)
- City University of New York (US)
- Canadian Institute for Advanced Research (CA)
- Alex's Lemonade Stand Foundation (US)
- University of British Columbia (CA)
- European Bioinformatics Institute (GB)
- Stanford Medicine (US)
- ORCID
- Haibe-Kains, Benjamin
- Adam, George Alexandru
- Hosny, Ahmed
- Khodakarami, Farnoosh
- Kusko, Rebecca
- Sansone, Susanna-Assunta
- Tong, Weida
- Mason, Christopher E.
- Jones, Wendell
- Furlanello, Cesare
- Waldron, Levi
- Wang, Bo
- McIntosh, Chris
- Goldenberg, Anna
- Kundaje, Anshul
- Greene, Casey S.
- Broderick, Tamara
- Hoffman, Michael M.
- Leek, Jeffrey T.
- Korthauer, Keegan
- Huber, Wolfgang
- Brazma, Alvis
- Pineau, Joelle
- Tibshirani, Robert
- Hastie, Trevor
- Ioannidis, John P. A.
- Quackenbush, John
- Aerts, Hugo J. W. L.
- Venue
- Nature
- Published
- 2020-10-14
- Volume
- 586
- Issue
- 7829
- Pages
- E14-E16
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC8144864
- OpenAlex
- W3008058947
- Cited by
- 540
- References
- 18
- Citations by year
- 2026: 83 · 2025: 91 · 2024: 97 · 2023: 110 · 2022: 79 · 2021: 70 · 2020: 10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RISalud Institutional Health Repository of Andalucía (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Biotechnology and Biological Sciences Research Council (BB/L005069/2, BB/E025080/1, BB/L005069/1, BB/J020265/1, BB/L005050/1)
- National Cancer Institute (U24CA180996, R01CA237170, CA237170)
- National Institute of General Medical Sciences (DP2GM123485)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducibility, explainability, deep learning, backtesting, generalizability, machine learning in healthcare, intellectual and developmental disabilities
- MeSH terms
- Algorithms, Artificial Intelligence, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
