The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Transparency and reproducibility in artificial intelligence
Open accessAbstract
Open Access articles citing this article. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications Kenniy Olorunnimbe & Herna Viktor Artificial Intelligence Review Open Access 30 June 2022 Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening Jenny Yang , Andrew A. S. Soltan & David A. Clifton npj Digital Medicine Open Access 07 June 2022 Bringing machine learning to research on intellectual and developmental disabilities: taking inspiration from neurological diseases Chirag Gupta , Pramod Chandrashekar … Daifeng Wang Journal of Neurodevelopmental Disorders Open Access 02 May 2022
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc. 39,40.”
From Artificial intelligence in pancreatic cancer (2022) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“highlighting the risk of using corporate models in a sensitive area such as medicine, where patient privacy, transparency and reliability are essential74,76”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication -including associations of reproducibility and rigour [21, 23]- without actually doing the work.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [23]
In the citing paper
“One of the most effective methods of mitigating AI biases is to ensure the use of diverse and representative datasets during AI development and training [67, 87].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [87]
In the citing paper
“A critical component of AI evaluation and development more generally is ensuring transparency and reproducibility, allowing independent research groups to review and build on prior scientific progress (196).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [196]
In the citing paper
“In addition, the lack of transparency and reproducibility of AI computational algorithm methods and code in many published studies threatens scientific progress and discovery, which require that independent scientists are able to scrutinize and reproduce results98.”
From Smart wearable devices in cardiovascular care: Where we are and how to move forward (2021) · cites this paper as [98]
In the citing paper
“Because of the particular significance of addressing the data-related concerns, different countries have developed policies and laws [160].”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [160]
In the citing paper
“Unfortunately, though, the majority of existing AI/ML models remain in specialized journals, with poorly maintained code and variable rigor in validation.283”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [283]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
(2015) · neural information processing systems · Cited by 912
A curated mammography data set for use in computer-aided detection and diagnosis research
Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017) · Scientific Data · doi:10.1038/sdata.2017.177 · Cited by 859
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 573
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · cited 1×
“Even with extensive description, reproducing complex computational pipelines based purely on text is a subjective and challenging task 9 .”
Background
Research works printed but not linked: 1
- Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).
Cited by
Sorted by citations
Crosby, D., Bhatia, S., Brindle, K. M., Coussens, L. M., Dive, C., Emberton, M., Esener, S., Fitzgerald, R. C., Gambhir, S. S., Kuhn, P., et al. (2022) · Science · doi:10.1126/science.aay9040 · Cited by 1,379
“This is an area where there is an opportunity for AI and ML to help detect cancer (114) and to predict risk of progression (115), although issues of transparency and reproducibility must be addressed (116).”
Smart wearable devices in cardiovascular care: Where we are and how to move forward
Bayoumy, K., Gaber, M., Elshafeey, A., Mhaimeed, O., Dineen, E. H., Marvel, F. A., Martin, S. S., Muse, E. D., Turakhia, M. P., Tarakji, K. G., et al. (2021) · Nature Reviews Cardiology · doi:10.1038/s41569-021-00522-7 · Cited by 829
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
“However, a recent investigation of one of these studies 130 found that significant information needed to actually reproduce the study was missing, greatly reducing the impact and ability of the field at large to scrutinize the study 131 and improve upon it.”
BackgroundLekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
Swanson, K., Wu, E., Zhang, A., Alizadeh, A. A., & Zou, J. (2023) · Cell · doi:10.1016/j.cell.2023.01.035 · Cited by 597
Fairness of artificial intelligence in healthcare: review and recommendations
Ueda, D., Kakinuma, T., Fujita, S., Kamagata, K., Fushimi, Y., Ito, R., Matsui, Y., Nozaki, T., Nakaura, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01474-3 · Cited by 607
Kocak, B., Baessler, B., Bakas, S., Cuocolo, R., Fedorov, A., Maier-Hein, L., Mercaldo, N., Müller, H., Orlhac, F., Pinto dos Santos, D., et al. (2023) · Insights into Imaging · doi:10.1186/s13244-023-01415-8 · Cited by 471
Thinking responsibly about responsible AI and ‘the dark side’ of AI
Mikalef, P., Conboy, K., Lundström, J. E., & Popovič, A. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2022.2026621 · Cited by 435
“…regarding the dark side implications it can potentially result in. Notable examples include the potential to increase bias and inequality (Akter et al., 2021; Luengo-Oroz et al., 2021), a lack of transparency (Haibe-Kains et al., 2020), and reduced human agency and freedom (Floridi & Cowls, 2019).”
BackgroundObtaining genetics insights from deep learning via explainable artificial intelligence
Novakovsky, G., Dexter, N., Libbrecht, M. W., Wasserman, W. W., & Mostafavi, S. (2022) · Nature Reviews Genetics · doi:10.1038/s41576-022-00532-2 · Cited by 415
Explainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372
Feng, L., Liu, Z., Li, C., Li, Z., Lou, X., Shao, L., Wang, Y., Huang, Y., Chen, H., Pang, X., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00215-6 · Cited by 339
Rodríguez-Espíndola, O., Chowdhury, S., Dey, P. K., Albores, P., & Emrouznejad, A. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.121562 · Cited by 279
Ahmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al-Fuqaha, A. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2021.100452 · Cited by 291
“Also, the Google Health study, published in Nature, which argued that an AI system can outperform radiologists at predicting cancer, was said to violate transparency and reproducibility [319, 320, 321].”
BackgroundThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Nam, D., Chapiro, J., Paradis, V., Seraphin, T. P., & Kather, J. N. (2022) · JHEP Reports · doi:10.1016/j.jhepr.2022.100443 · Cited by 244
Accelerating antibiotic discovery through artificial intelligence
Melo, M. C. R., Maasch, J. R. M. A., & de la Fuente-Nunez, C. (2021) · Communications Biology · doi:10.1038/s42003-021-02586-0 · Cited by 234
“However, standards of openness and reproducibility in biomedical ML are still subject to debate 126 , and some argue that AI generally suffers from a reproducibility crisis, not unlike that of psychology 127 .”
BackgroundArtificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196
Yang, J., Soltan, A. A. S., & Clifton, D. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00614-9 · Cited by 181
“Furthermore, the sharing of code and data is widely seen as being crucial to reproducibility; however, in many medical fields, they are rarely made available, as patient health information is highly regulated due to privacy concerns (Haibe-Kains et al., 2020).”
BackgroundKourou, K. D., Exarchos, K., Papaloukas, C. L., Sakaloglou, P., Exarchos, T., & Fotiadis, D. I. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.006 · Cited by 175
“Despite this, the enormous promise of AI in cancer research should be carefully addressed alongside answers to the challenges of transparency and reproducibility [9–11].”
BackgroundWhere Medical Statistics Meets Artificial Intelligence
Hunter, D. J., & Holmes, C. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2212850 · Cited by 169
The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI
Birch, J. (2024) · doi:10.1093/9780191966729.001.0001 · Cited by 236
Molecular de-extinction of ancient antimicrobial peptides enabled by machine learning
Maasch, J. R. M. A., Torres, M. D. T., Melo, M. C. R., & de la Fuente‐Núñez, C. (2023) · Cell Host & Microbe · doi:10.1016/j.chom.2023.07.001 · Cited by 161
AI and the Future of Skills, Volume 1
OECD (2021) · Educational research and innovation · doi:10.1787/5ee71f34-en · Cited by 233
Materials Acceleration Platforms: On the way to autonomous experimentation
Flores‐Leonar, M. M., Mejía-Mendoza, L. M., Aguilar‐Granda, A., Sánchez-Lengeling, B., Tribukait, H., Amador‐Bedolla, C., & Aspuru‐Guzik, A. (2020) · Current Opinion in Green and Sustainable Chemistry · doi:10.1016/j.cogsc.2020.100370 · Cited by 155
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J. R., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 255
“Conversely, if the code and data for reproducing all results in a study are not available, then identifying the precise sources of errors in a study becomes hard ( 79 – 81 ).”
Franco D’Souza, R., Mathew, M., Mishra, V., & Surapaneni, K. M. (2024) · Medical Education Online · doi:10.1080/10872981.2024.2330250 · Cited by 147
“Transparent AI systems enable the comprehension of the logic behind recommendations and assist in the continuous improvement of educational content [9].”
BackgroundWaqas, A., Bui, M. M., Glassy, E. F., Naqa, I. E., Borkowski, P., Borkowski, A., & Rasool, G. (2023) · Laboratory Investigation · doi:10.1016/j.labinv.2023.100255 · Cited by 158
Trustworthy AI and robotics: Implications for the AEC industry
Emaminejad, N., & Akhavian, R. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104298 · Cited by 158
“In AIR applications, transparency is often related to the concept of interpretability where the operations of a system can be understood by a human through introspection or explanation (Biran and Cotton 2017, Haibe-Kains, Adam et al. 2020).”
BackgroundDeep learning in the stock market—a systematic survey of practice, backtesting, and applications
Olorunnimbe, K., & Viktor, H. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10226-0 · Cited by 170
Chowdhury, S., Joel-Edgar, S., Dey, P. K., Bhattacharya, S., & Kharlamov, A. (2023) · The International Journal of Human Resource Management · doi:10.1080/09585192.2022.2066981 · Cited by 140
“In this context, transparency within AI systems is intangible because it represents the system’s quality and characteristics (Haibe-Kains et al., 2020).”
BackgroundArtificial intelligence in pancreatic cancer
Huang, B., Huang, H., Zhang, S., Zhang, D., Shi, Q., Liu, J., & Guo, J. (2022) · Theranostics · doi:10.7150/thno.77949 · Cited by 142
Ethical considerations on artificial intelligence in dentistry: A framework and checklist
Rokhshad, R., Ducret, M., Chaurasia, A., Karteva, T., Radenković, M., Roganović, J., Hamdan, M. H., Mohammad‐Rahimi, H., Krois, J., Lahoud, P., et al. (2023) · Journal of Dentistry · doi:10.1016/j.jdent.2023.104593 · Cited by 149
Digital Phenotyping: Data-Driven Psychiatry to Redefine Mental Health
Oudin, A., Maatoug, R., Bourla, A., Ferreri, F., Bonnot, O., Millet, B., Schoeller, F., Mouchabac, S., & Adrien, V. (2023) · Journal of Medical Internet Research · doi:10.2196/44502 · Cited by 156
Sampat, B. H., Mogaji, E., & Nguyen, N. P. (2023) · International Journal of Bank Marketing · doi:10.1108/ijbm-07-2022-0328 · Cited by 165
The current status of risk-stratified breast screening
Clift, A. K., Dodwell, D. J., Lord, S., Petrou, S., Brady, M., Collins, P. G. S., & Hippisley‐Cox, J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01550-3 · Cited by 142
Fehr, J., Citro, B., Malpani, R., Lippert, C., & Madai, V. I. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1267290 · Cited by 165
Crossnohere, N. L., Elsaid, M. I., Paskett, J., Bose‐Brill, S., & Bridges, J. F. P. (2022) · Journal of Medical Internet Research · doi:10.1016/j.jval.2022.04.1285 · Cited by 148
“Reproducibility is a basic tenet of good scientific practice [44].”
Hickman, S. E., Woitek, R., Le, E., Im, Y. R., Luxhøj, C. M., Avilés-Rivero, A. I., Baxter, G. C., Mackay, J., & Gilbert, F. J. (2021) · Radiology · doi:10.1148/radiol.2021210391 · Cited by 133
Tsopra, R., Fernández, X. M., Luchinat, C., Alberghina, L., Lehrach, H., Vanoni, M., Dreher, F., Sezerman, O. U., Marc Cuggia, A., de Tayrac, M., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01634-3 · Cited by 122
Promises and perils of artificial intelligence in dentistry
Pethani, F. (2020) · Australian Dental Journal · doi:10.1111/adj.12812 · Cited by 129
Wang, Y.-Q., Wang, H., Song, Y., Zhou, S., Li, Q., Liang, B., Liu, W., Zhao, Y.-W., & Wang, A. (2023) · Water Research · doi:10.1016/j.watres.2023.120676 · Cited by 119
Artificial Intelligence in CT and MR Imaging for Oncological Applications
Paudyal, R., Shah, A. D., Akın, O., Gian, R. K., Konar, A. S., Hatzoglou, V., Mahmood, U. A., Lee, N. Y., Wong, R. J., Banerjee, S., et al. (2023) · Cancers · doi:10.3390/cancers15092573 · Cited by 113
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111
“Despite strong evidence of the scientific and engineering benefits of open research practices, openness is not a given in machine learning and AI research [18, 20, 30].”
BackgroundIs AI leading to a reproducibility crisis in science?
Ball, P. (2023) · Nature · doi:10.1038/d41586-023-03817-6 · Cited by 106
Rethink reporting of evaluation results in AI
Burnell, R., Schellaert, W., Burden, J., Ullman, T. D., Martinez-Plumed, F., Tenenbaum, J. B., Rutar, D., Cheke, L. G., Sohl-Dickstein, J., Mitchell, M., et al. (2023) · Science · doi:10.1126/science.adf6369 · Cited by 104
Advancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 121
Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use
Windecker, D., Baj, G., Shiri, I., Kazaj, P. M., Kaesmacher, J., Gräni, C., & Siontis, G. C. M. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.8052 · Cited by 127
Javaheri, T., Homayounfar, M., Amoozgar, Z., Reiazi, R., Homayounieh, F., Abbas, E., Laali, A., Radmard, A. R., Gharib, M. H., Mousavi, S. A., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00399-3 · Cited by 129
Self-citationDeep learning technology for construction machinery and robotics
You, K., Zhou, C., & Ding, L. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104852 · Cited by 104
Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development
Bradshaw, T., Boellaard, R., Dutta, J., Jha, A. K., Jacobs, P. D., Li, Q., Liu, C., Sitek, A., Saboury, B., Scott, P. J. H., et al. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.121.262567 · Cited by 103
“There are also valid concerns about the reproducibility of claims made in many published AI studies (2) and the generalizability of trained algorithms (3).”
BackgroundGao, Y., Ventura‐Díaz, S., Wang, X., He, M., Xu, Z., Weir, A., Zhou, H.-Y., Zhang, T., van Duijnhoven, F. H., Han, L., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-53450-8 · Cited by 93
“To address the challenges of transparency and reproducibility in the application of arti fi cial intelligence in medicine 46 , we followed established guidelines 47,48 , which provide best practices for study protocols and reporting speci fi c to machine learning.”
Visvikis, D., Lambin, P., Mouridsen, K., Hustinx, R., Laßmann, M., Rischpler, C., Shi, K., & Pruim, J. (2022) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-022-05891-w · Cited by 100
“At the same time, AI results are being criticized because of the lack of transparency and consequently a potential lack of reproducibility [7].”
BackgroundRequirements and reliability of AI in the medical context
Balagurunathan, Y., Mitchell, R., & Naqa, I. E. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.02.024 · Cited by 94
“One such concern has been widely referenced in a recent effort to improve transparency and reliability of these deep network findings [102], which has outlined the need for open disclosure of codebase, datasets used for training and preprocessing steps to achieve desired outcome.”
BackgroundTowards Risk‐Free Trustworthy Artificial Intelligence: Significance and Requirements
Alzubaidi, L., Al-Sabaawi, A., Bai, J., Dukhan, A., Alkenani, A. H., Al‐Asadi, A., Alwzwazy, H. A., Manoufali, M., Fadhel, M. A., Albahri, A. S., et al. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/4459198 · Cited by 116
The impact of commercial health datasets on medical research and health-care algorithms
Alberto, I. R. I., Alberto, N. R. I., Ghosh, A. K., Jain, B., Jayakumar, S., Martinez-Martin, N., McCague, N., Moukheiber, D., Moukheiber, L., Moukheiber, M., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00025-0 · Cited by 86
“28 Although in this case access to the codebase was also pointed out as an issue, the point remains that restricted-access datasets have made reproducibility and generalisability in machine learning for health particularly challenging.”
BackgroundSaghiri, A. M., Vahidipour, S. M., Jabbarpour, M. R., Sookhak, M., & Forestiero, A. (2022) · Applied Sciences · doi:10.3390/app12084054 · Cited by 122
Explainable Artificial Intelligence (XAI)
Ridley, M. (2022) · Information Technology and Libraries · doi:10.6017/ital.v41i2.14683 · Cited by 115
“48”
Artificial Intelligence in Medicine and Dentistry
Vodanović, M., Subašić, M., Milošević, D., & Pavičin, I. S. (2023) · Acta Stomatologica Croatica · doi:10.15644/asc57/1/8 · Cited by 88
Is there a role for statistics in artificial intelligence?
Friedrich, S., Antes, G., Behr, S., Binder, H., Brannath, W., Dumpert, F., Ickstadt, K., Kestler, H. A., Lederer, J., Leitgöb, H., et al. (2021) · Advances in Data Analysis and Classification · doi:10.1007/s11634-021-00455-6 · Cited by 99
brainlife.io: a decentralized and open-source cloud platform to support neuroscience research
Hayashi, S., Caron, B. A., Heinsfeld, A. S., Vinci‐Booher, S., McPherson, B., Bullock, D. N., Bertò, G., Niso, G., Hanekamp, S., Levitas, D. J., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02237-2 · Cited by 84
“13,22,23 This is largely because small sample sizes increase the probability of reporting spurious effects as statistically significant.”
BackgroundRadiomics and machine learning applications in rectal cancer: Current update and future perspectives
Stanzione, A., Verde, F., Romeo, V., Boccadifuoco, F., Mainenti, P. P., & Maurea, S. (2021) · World Journal of Gastroenterology · doi:10.3748/wjg.v27.i32.5306 · Cited by 74
Friedrich, S., Groß, S., König, I. R., Engelhardt, S., Bahls, M., Heinz, J., Huber, C., Kaderali, L., Kelm, M., Leha, A., et al. (2021) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztab054 · Cited by 71
Artificial Intelligence in Drug Design: Algorithms, Applications, Challenges and Ethics
Arabi, A. A. (2021) · Future Drug Discovery · doi:10.4155/fdd-2020-0028 · Cited by 75
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68
Automated chart filing on panoramic radiographs using deep learning
Vinayahalingam, S., Goey, R., Kempers, S., Schoep, J., Cherici, T., Moin, D. A., & Hanisch, M. R. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103864 · Cited by 69
Sharing biological data: why, when, and how
Wilson, S. L., Way, G. P., Bittremieux, W., Armache, J., Haendel, M. A., & Hoffman, M. M. (2021) · FEBS Letters · doi:10.1002/1873-3468.14067 · Cited by 66
Self-citationRethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 123
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication —in-cluding associations of reproducibility and rigour [21, 23]— without actually doing the work.”
BackgroundOn the role of artificial intelligence in medical imaging of COVID-19
Born, J., Beymer, D., Rajan, D., Coy, A., Mukherjee, V. V., Manica, M., Prasanna, P. M., Ballah, D., Guindy, M., Shaham, D., et al. (2021) · Patterns · doi:10.1016/j.patter.2021.100269 · Cited by 73
Harfouche, A., Quinio, B., Saba, M., & Saba, P. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10352-8 · Cited by 55
“Nevertheless, by significantly reducing and sometimes eliminating the reliance on domain-expert, these new data-driven approaches have raised concerns about their models' transparency (Haibe-Kains et al., 2020) and interpretability (Lipton, 2018).”
BackgroundReproducible Research: A Retrospective
Peng, R. D., & Hicks, S. C. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-012420-105110 · Cited by 78
Chow, R., Midroni, J., Kaur, J., Boldt, G., Liu, G., Eng, L., Liu, F.-F., Haibe-Kains, B., Lock, M., & Raman, S. (2023) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djad013 · Cited by 65
Self-citationFrom Machine Learning to Patient Outcomes: A Comprehensive Review of AI in Pancreatic Cancer
Tripathi, S., Tabari, A., Mansur, A., Dabbara, H., Bridge, C. P., & Daye, D. (2024) · Diagnostics · doi:10.3390/diagnostics14020174 · Cited by 70
Gupta, C., Chandrashekar, P. B., Jin, T., He, C., Khullar, S., Chang, Q., & Wang, D. (2022) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-022-09438-w · Cited by 66
Loop-Mediated Isothermal Amplification Detection of SARS-CoV-2 and Myriad Other Applications
Moore, K. J. M., Cahill, J., Aidelberg, G., Aronoff, R., Bektaş, A., Bezdan, D., Butler, D., Chittur, S. V., Codyre, M., Federici, F., et al. (2021) · Journal of Biomolecular Techniques JBT · doi:10.7171/jbt.21-3203-017 · Cited by 59
Self-citationBullock, G. S., Hughes, T., Arundale, A. J. H., Ward, P., Collins, P. G. S., & Kluzek, S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01655-6 · Cited by 61
“…coefficients including the intercept) [8] or the complete code and tuned hyperparam-eters (e.g., for machine learning and deep learning models) [3, 46] can allow the creation of easy to use applications, where sports medicine practitioners can input relevant data to calculate an individual…”
BackgroundScheda, R., & Diciotti, S. (2022) · Applied Sciences · doi:10.3390/app12136681 · Cited by 60
Guidelines and standard frameworks for artificial intelligence in medicine: a systematic review
Shiferaw, K. B., Roloff, M., Balaur, I., Welter, D., Waltemath, D., & Zeleke, A. A. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae155 · Cited by 57
“The reproducibility in AI builds trust in the developed models and results [14, 40].”
BackgroundThe AIMe registry for artificial intelligence in biomedical research
Matschinske, J., Alcaraz, N., Benis, A., Golebiewski, M., Grimm, D. G., Heumos, L., Kacprowski, T., Lazareva, O., List, M., Louadi, Z., et al. (2021) · Nature Methods · doi:10.1038/s41592-021-01241-0 · Cited by 54
Sacco, A., Mandrile, L., Tay, L., Itoh, N., Raj, A., Moure, A., del Campo, A., Fernández, J. F., Paton, K. R., Wood, S., et al. (2023) · Metrologia · doi:10.1088/1681-7575/acf76d · Cited by 53
“It lacks both reference artifacts and operating procedures for quantitative analyses, including the standardization of data analysis routines that include machine learning and/or multivariate modeling—on whose operative details often scarce information is provided and no reproducibility effort is spent [11].”
On the Commoditization of Artificial Intelligence
Abonamah, A. A., Tariq, M. U., & Shilbayeh, S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.696346 · Cited by 69
Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence
Duran-Frigola, M., Cigler, M., & Winter, G. E. (2023) · Journal of the American Chemical Society · doi:10.1021/jacs.2c11098 · Cited by 50
Coiera, E. W., & Liu, S. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100860 · Cited by 56
Duong‐Trung, N., Born, S., Kim, J. W., Schermeyer, M., Paulick, K., Borisyak, M., Bournazou, M. N. C., Werner, T., Scholz, R., Schmidt-Thieme, L., et al. (2022) · Biochemical Engineering Journal · doi:10.1016/j.bej.2022.108764 · Cited by 56
Biological research and self-driving labs in deep space supported by artificial intelligence
Sanders, L., Scott, R. T., Yang, J. H., Qutub, A. A., Martín, H. G., Berrios, D. C., Hastings, J. J. A., Rask, J. C., Mackintosh, G., Hoarfrost, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00618-4 · Cited by 59
“Additional work is needed to establish widely adopted standards for AI readiness in these research domains 103,104 .”
BackgroundSelf-citationGeneralizability of an acute kidney injury prediction model across health systems
Cao, J., Zhang, X., Shahinian, V. B., Yin, H., Steffick, D., Saran, R., Crowley, S. T., Mathis, M. R., Nadkarni, G. N., Heung, M., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00563-8 · Cited by 56
Murray, J. D., Lange, J. J., Bennett-Lenane, H., Holm, R., Kuentz, M., O’Dwyer, P. J., & Griffin, B. T. (2023) · European Journal of Pharmaceutical Sciences · doi:10.1016/j.ejps.2023.106562 · Cited by 52
Kazmierski, M., Welch, M. L., Kim, S., McIntosh, C., Rey‐McIntyre, K., Huang, S. H., Patel, T., Tadic, T., Milosevic, M., Liu, F., et al. (2023) · Cancer Research Communications · doi:10.1158/2767-9764.crc-22-0152 · Cited by 46
Self-citationPetrick, N., Chen, W., Delfino, J. G., Gallas, B. D., Kang, Y., Krainak, D. M., Sahiner, B., & Samala, R. K. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.5.051804 · Cited by 55
“One example of this is a lack of code and data sharing hindering the ability of outside groups to perform reproducibility studies and assessments for many AI/ML algorithms appearing in the peer-reviewed literature.(78) Therefore, there remains transparency challenges including determining exactly what information is n…”
BackgroundGhislat, G., Hernández-Hernández, S., Piyawajanusorn, C., & Ballester, P. J. (2024) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2024.2403639 · Cited by 53
Imrie, F., Cebere, B., McKinney, E. F., & van der Schaar, M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000276 · Cited by 53
Intra-oral scan segmentation using deep learning
Vinayahalingam, S., Kempers, S., Schoep, J., Hsu, T.-M. H., Moin, D. A., van Ginneken, B., Flügge, T. V., Hanisch, M. R., & Xi, T. (2023) · BMC Oral Health · doi:10.1186/s12903-023-03362-8 · Cited by 45
Stewart, J. E., Lü, J., Goudie, A., Bennamoun, M., Sprivulis, P., Sanfilippo, F. M., & Dwivedi, G. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252612 · Cited by 47
Artificial intelligence for the next generation of precision oncology
Ballester, P. J., & Carmona, J. (2021) · npj Precision Oncology · doi:10.1038/s41698-021-00216-w · Cited by 51
“…detail on the development of algorithms, and encouraging sharing of data sets and code to enhance the transparency in the reporting of AI algorithms in medicine 11,12,24,25 and allow other researchers to determine the rigor, quality, reproducibility, and generalizability of the fi ndings 26 – 28 .”
BackgroundIs regulatory science ready for artificial intelligence?
Härtung, T., Whelan, M. P., Tong, W., & Califf, R. M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01596-0 · Cited by 44
“It has been argued that the lack of AI reproducibility hinders trust in AI application 8 .”
Self-citationAddendum: International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A. W., et al. (2020) · Nature · doi:10.1038/s41586-020-2679-9 · Cited by 42
Challenging presumed technological superiority when working with (artificial) colleagues
Rieger, T., Roesler, E., & Manzey, D. (2022) · Scientific Reports · doi:10.1038/s41598-022-07808-x · Cited by 45
“This is important as even though classical decision support systems (DSS) and AI share many commonalities (e.g., the changed human role 7 or the lack of systems transparency 1,12,13 ), there might also be some key differences in the perception of those systems.”
BackgroundHan, Y., Ceross, A., & Bergmann, J. (2024) · JMIR AI · doi:10.2196/46871 · Cited by 40
Metadata
- Authors
- Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., Mason, C. E., Jones, W., Dopazo, J., Furlanello, C., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C. S., Broderick, T., Hoffman, M. M., Leek, J. T., Korthauer, K., Huber, W., Brazma, A., Pineau, J., Tibshirani, R., Hastie, T., Ioannidis, J. P. A., Quackenbush, J., & Aerts, H. J. W. L.
- Affiliations
- University of Toronto (CA)
- Brigham and Women's Hospital (US)
- University Health Network (CA)
- National Center for Toxicological Research (US)
- University of Oxford (GB)
- SAS Institute (United States) (US)
- Q2 Solutions (United States) (US)
- City University of New York (US)
- Canadian Institute for Advanced Research (CA)
- Alex's Lemonade Stand Foundation (US)
- University of British Columbia (CA)
- European Bioinformatics Institute (GB)
- Stanford Medicine (US)
- ORCID
- Haibe-Kains, Benjamin
- Adam, George Alexandru
- Hosny, Ahmed
- Khodakarami, Farnoosh
- Kusko, Rebecca
- Sansone, Susanna-Assunta
- Tong, Weida
- Mason, Christopher E.
- Jones, Wendell
- Furlanello, Cesare
- Waldron, Levi
- Wang, Bo
- McIntosh, Chris
- Goldenberg, Anna
- Kundaje, Anshul
- Greene, Casey S.
- Broderick, Tamara
- Hoffman, Michael M.
- Leek, Jeffrey T.
- Korthauer, Keegan
- Huber, Wolfgang
- Brazma, Alvis
- Pineau, Joelle
- Tibshirani, Robert
- Hastie, Trevor
- Ioannidis, John P. A.
- Quackenbush, John
- Aerts, Hugo J. W. L.
- Venue
- Nature
- Published
- 2020-10-14
- Volume
- 586
- Issue
- 7829
- Pages
- E14-E16
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC8144864
- OpenAlex
- W3008058947
- Cited by
- 540
- References
- 18
- Citations by year
- 2026: 83 · 2025: 91 · 2024: 97 · 2023: 110 · 2022: 79 · 2021: 70 · 2020: 10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RISalud Institutional Health Repository of Andalucía (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Biotechnology and Biological Sciences Research Council (BB/L005069/2, BB/E025080/1, BB/L005069/1, BB/J020265/1, BB/L005050/1)
- National Cancer Institute (U24CA180996, R01CA237170, CA237170)
- National Institute of General Medical Sciences (DP2GM123485)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducibility, explainability, deep learning, backtesting, generalizability, machine learning in healthcare, intellectual and developmental disabilities
- MeSH terms
- Algorithms, Artificial Intelligence, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
