The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Transparency and reproducibility in artificial intelligence
Open accessAbstract
Open Access articles citing this article. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications Kenniy Olorunnimbe & Herna Viktor Artificial Intelligence Review Open Access 30 June 2022 Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening Jenny Yang , Andrew A. S. Soltan & David A. Clifton npj Digital Medicine Open Access 07 June 2022 Bringing machine learning to research on intellectual and developmental disabilities: taking inspiration from neurological diseases Chirag Gupta , Pramod Chandrashekar … Daifeng Wang Journal of Neurodevelopmental Disorders Open Access 02 May 2022
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, AI's transparency and interpretability are challenged by factors such as patient privacy, algorithm interpretability, publication bias, etc. 39,40.”
From Artificial intelligence in pancreatic cancer (2022) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“highlighting the risk of using corporate models in a sensitive area such as medicine, where patient privacy, transparency and reliability are essential74,76”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [76]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication -including associations of reproducibility and rigour [21, 23]- without actually doing the work.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [23]
In the citing paper
“One of the most effective methods of mitigating AI biases is to ensure the use of diverse and representative datasets during AI development and training [67, 87].”
From Fairness of artificial intelligence in healthcare: review and recommendations (2024) · cites this paper as [87]
In the citing paper
“A critical component of AI evaluation and development more generally is ensuring transparency and reproducibility, allowing independent research groups to review and build on prior scientific progress (196).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [196]
In the citing paper
“In addition, the lack of transparency and reproducibility of AI computational algorithm methods and code in many published studies threatens scientific progress and discovery, which require that independent scientists are able to scrutinize and reproduce results98.”
From Smart wearable devices in cardiovascular care: Where we are and how to move forward (2021) · cites this paper as [98]
In the citing paper
“Because of the particular significance of addressing the data-related concerns, different countries have developed policies and laws [160].”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [160]
In the citing paper
“Unfortunately, though, the majority of existing AI/ML models remain in specialized journals, with poorly maintained code and variable rigor in validation.283”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [283]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
(2015) · neural information processing systems · Cited by 912
A curated mammography data set for use in computer-aided detection and diagnosis research
Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017) · Scientific Data · doi:10.1038/sdata.2017.177 · Cited by 859
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 573
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 464
Self-citationBluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationState of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · cited 1×
“Even with extensive description, reproducing complex computational pipelines based purely on text is a subjective and challenging task 9 .”
Background
Research works printed but not linked: 1
- Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).
Cited by
Sorted by citations
Items 201–300 of 541
Statistical Perspectives on Reproducibility: Definitions and Challenges
Simkus, A., Coolen‐Maturi, T., Coolen, F. P. A., & Bendtsen, C. (2025) · Journal of Statistical Theory and Practice · doi:10.1007/s42519-025-00459-x · Cited by 14
“Haibe-Kains et al. [67] suggest that if data cannot be shared, an independent, highly trained investigator should verify the analysis.”
Gonzalez, J. A. H., & Colom, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1491823 · Cited by 20
Cognitive Modeling Using Artificial Intelligence
Frank, M. C., & Goodman, N. D. (2025) · Annual Review of Psychology · doi:10.1146/annurev-psych-030625-040748 · Cited by 24
Generating Small Areal Synthetic Microdata from Public Aggregated Data Using an Optimization Method
Lin, Y., & Xiao, N. (2023) · The Professional Geographer · doi:10.1080/00330124.2023.2207640 · Cited by 12
“All scientific methods have their limitations, and not being able to replicate them prevents us from fully understanding the limitations and making improvements (P ’ erignon et al. 2019; Haibe-Kains et al. 2020).”
BackgroundArtificial Intelligence in the Screening, Diagnosis, and Management of Aortic Stenosis
Zhang, Y., Wang, M., Zhang, E., & Wu, Y. (2024) · Reviews in Cardiovascular Medicine · doi:10.31083/j.rcm2501031 · Cited by 13
3D printed electro-responsive system with programmable drug release
Alkahtani, M. E., Sun, S., Chapman, C. A. R., Gaisford, S., Orlu, M., Elbadawi, M., & Basit, A. W. (2024) · Materials Today Advances · doi:10.1016/j.mtadv.2024.100509 · Cited by 12
Dai, L., Yin, J., Xin, X., Yao, C., Tang, Y., Xia, X., Chen, Y., Lai, S., Lu, G., Huang, J., et al. (2025) · Cancer Imaging · doi:10.1186/s40644-025-00855-3 · Cited by 12
“Although ML predictive models have been reported to be powerful [33, 38–43], they are often referred to as black boxes because they lack interpretability and trans-parency [44].”
Nicol, E. D., Weir-McCall, J. R., Shaw, L. J., & Williamson, E. E. (2022) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2022.07.005 · Cited by 12
Reproducibility in Machine Learning for Medical Imaging
Olivier Colliot, A., Thibeau–Sutre, E., & Burgos, N. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_21 · Cited by 16
Ostropolets, A., Albogami, Y., Conover, M. M., Banda, J. M., Baumgartner, W. A., Blacketer, C., Desai, P., DuVall, S. L., Fortin, S., Gilbert, J. P., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad009 · Cited by 12
FAIR AI models in high energy physics
Duarte, J. M., Li, H., Roy, A., Zhu, R., Huerta, E. A., Díaz, D., Harris, P. C., Kansal, R., Katz, D. S., Kavoori, I. H., et al. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad12e3 · Cited by 17
“…integral to the 2012 discovery of the Higgs boson [1, 2] and subsequent observation of its decay to bottom quarks [3, 4] at the CERN Large Hadron Collider (LHC), where they were used to identify in proton–proton collisions the nature and origin of ‘jets’ of particles produced in the collisions.”
BackgroundTen simple rules for building and maintaining a responsible data science workflow
Stoudt, S., Jernite, Y., Marshall, B., Marwick, B., Sharan, M., Whitaker, K., & Danchev, V. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1012232 · Cited by 11
Kazerooni, A. F., Familiar, A., Aboian, M., Brüningk, S. C., Vossough, A., Linguraru, M. G., Huang, R. Y., Hargrave, D. R., Peet, A. C., Resnick, A. C., et al. (2025) · The Lancet Oncology · doi:10.1016/s1470-2045(25)00489-9 · Cited by 14
Gundersen, O. E., Shamsaliei, S., Kjærnli, H. S., & Helge Langseth, A. (2023) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3589806.3600044 · Cited by 17
“Given the prevalence of deep learning [3], this is a severe issue for research in artificial intelligence that is recognized by the community [12, 16, 35].”
Backgroundvan Kampen, A. H. C., Mahamune, U., Jongejan, A., van Schaik, B. D. C., Balashova, D., Lashgari, D., Pras‐Raves, M. L., Wever, E. J. M., Dane, A. D., García-Valiente, R., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52446-8 · Cited by 21
“This was also recognized by the arti fi cial intelligence community, which started initiatives to develop speci fi c AI-oriented guidelines 101 – 103 .”
Zhao, D., Frey, N. C., McDonald, J., Hubbell, M., Bestor, D., Jones, M. P., Prout, A., Gadepally, V. N., & Samsi, S. (2022) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw55747.2022.00126 · Cited by 13
Artificial Intelligence Governance and Otolaryngology‐Head and Neck Surgery
Nwosu, O. I., Crowson, M. G., & Rameau, A. (2023) · The Laryngoscope · doi:10.1002/lary.31013 · Cited by 11
Artificial Intelligence Assisted Drug Research and Development
Soni, K., & Hasija, Y. (2022) · IEEE Delhi Section Conference (DELCON) · doi:10.1109/delcon54057.2022.9753179 · Cited by 11
“It is impossible to replicate AI-forecasted outcomes, which is a significant problem that should be solved immediately [53].”
BackgroundThe EU AI Liability Directive (AILD): Bridging Information Gaps
Ziosi, M., Mökander, J., Novelli, C., Casolari, F., Taddeo, M., & Floridi, L. (2023) · doi:10.2139/ssrn.4470725 · Cited by 12
Navigating the development challenges in creating complex data systems
Dittmer, S., Roberts, M. T., Gilbey, J., Biguri, A., SELBY, I. A., Breger, A., Thorpe, M., Weir-McCall, J. R., Gkrania‐Klotsas, E., Korhonen, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00665-x · Cited by 10
O’Connell, S., Cannon, D. M., & Broin, P. Ó. (2023) · Human Brain Mapping · doi:10.1002/hbm.26521 · Cited by 10
“Secondly, clinical decision systems require rigorous validation and reporting frameworks for more interpretable models; the use of opaque deep learning algorithms makes validation and transparency more difficult to achieve (Collins et al., 2015; Haibe-Kains et al., 2020).”
BackgroundData privacy-aware machine learning approach in pancreatic cancer diagnosis
Akmeşe, Ö. F. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02657-2 · Cited by 12
Feng, Y., & Chen, H. (2022) · Journal of Interactive Advertising · doi:10.1080/15252019.2022.2126337 · Cited by 10
“Notably, a team of scientists published an article in Nature to call for attention to the transparency and reproducibility of existing AI applications (Haibe-Kains et al. 2020).”
BackgroundMethodLearnChain: Transparent and cooperative reinforcement learning on Blockchain
Sami, H., Mizouni, R., Otrok, H., Singh, S., Bentahar, J., & Mourad, A. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.09.012 · Cited by 10
Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research
Subramonyam, H., & Hullman, J. R. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326591 · Cited by 10
Towards a framework for interoperability and reproducibility of predictive models
Rahrooh, A., Garlid, A. O., Bartlett, K., Coons, W. R., Petousis, P., Hsu, W. W. Y., & Bui, A. A. T. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104551 · Cited by 10
Artificial intelligence for microbial biotechnology: beyond the hype
Robinson, S. L. (2021) · Microbial Biotechnology · doi:10.1111/1751-7915.13943 · Cited by 10
The need for a prediction model assessment framework
Islam, S. M. S., & Khosravi, A. (2021) · The Lancet Global Health · doi:10.1016/s2214-109x(21)00022-x · Cited by 10
Medical Artificial Intelligence and Human Values
Manrai, A. K., Yu, K., & Kohane, I. S. (2024) · New England Journal of Medicine · doi:10.1056/nejmc2408971 · Cited by 10
mlf-core: a framework for deterministic machine learning
Heumos, L., Ehmele, P., Cuellar, L. K., Menden, K., Miller, E., Lemke, S., Gabernet, G., & Nahnsen, S. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad164 · Cited by 11
“The reasons for irrepro-ducible ML are diverse, including unavailable (raw) data, unpublished code, unreported hyperparameters, and missing reports on the hardware and software used for the analysis (Olorisade et al. 2017; Henderson et al. 2018; Haibe-Kains et al. 2020).”
Reproducing Popularity Bias in Recommendation: The Effect of Evaluation Strategies
Daniil, S., Cuper, M., Liem, C. C. S., van Ossenbruggen, J., & Laura Hollink, A. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3637066 · Cited by 9
“Therefore, lack of published code and data render the replication of existing work by other researchers challenging [18].”
BackgroundAttafi, O. A., Clementel, D., Kyritsis, K., Capriotti, E., Farrell, G., Fragkouli, S.-C., Castro, L. J., Hatos, A., Lenaerts, T., Mazurenko, S., et al. (2024) · GigaScience · doi:10.1093/gigascience/giae094 · Cited by 9
Mukhtar, A., Hadwiger, M., Wotawa, F., & Schweiger, G. (2025) · Energy and AI · doi:10.1016/j.egyai.2025.100658 · Cited by 11
How artificial intelligence shapes job anxiety: the mediating role of AI stress
Fang, L. B., Tang, L., & Yang, H. (2026) · Internet Research · doi:10.1108/intr-09-2024-1462 · Cited by 13
Amram, D., Cignoni, A., Banfi, T., & Ciuti, G. (2022) · Open Research Europe · doi:10.12688/openreseurope.14524.1 · Cited by 7
Gryska, E. A., Björkman‐Burtscher, I. M., Jakola, A. S., Dunås, T., Schneiderman, J. F., & Heckemann, R. A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059000 · Cited by 9
A Metamorphic Testing Framework and Toolkit for Modular Automated Driving Systems
Underwood, R., Luu, Q.-H., & Liu, H. (2023) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met59151.2023.00010 · Cited by 9
“This issue, being referred to as the reproducibility problem, arises from the nature of both hardware and software components used by the ADS, including deep learning algorithms that are commonly responsible for perceptions and tactical planning tasks [1], [20].”
BackgroundJackson, A., Hua, C., Olch, A. J., Yorke, E. D., Rancati, T., Milano, M. T., Constine, L. S., Marks, L. B., & Bentzen, S. M. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.02.018 · Cited by 9
Oncology Informatics: Status Quo and Outlook
Putora, P. M., Baudis, M., Beadle, B. M., Naqa, I. E., Giordano, F. A., & Nicolay, N. H. (2020) · Oncology · doi:10.1159/000507586 · Cited by 9
How is Artificial Intelligence Changing Science?
Deelman, E. (2023) · IEEE International Conference on eScience · doi:10.1109/e-science58273.2023.10254913 · Cited by 9
“Using AI as part of the computational portions of the data lifecycle makes reproducibility even more difficult [29].”
MethodThe NCI Imaging Data Commons as a platform for reproducible research in computational pathology
Schacherer, D. P., Herrmann, M. D., Clunie, D., Höfener, H., Clifford, W., Longabaugh, W. J. R., Pieper, S., Kikinis, R., Fedorov, A., & Homeyer, A. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107839 · Cited by 11
Luo, X., Chen, X., & Yao, Y. (2024) · Applied Sciences · doi:10.3390/app14020669 · Cited by 8
Secondary Use of Health Data for Medical AI: A Cross-Regional Examination of Taiwan and the EU
Ho, C. (2024) · Asian Bioethics Review · doi:10.1007/s41649-024-00279-4 · Cited by 10
“…it is imperative to establish more transparent data governance mechanisms that not only safeguard privacy but also empower data subjects with detailed insights into the precise nature of data usage, the entities involved, and the purpose behind such utilization (Haibe-Kains et al. 2020).”
BackgroundAskar, M., Tafavvoghi, M., Småbrekke, L., Bongo, L. A., & Svendsen, K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309175 · Cited by 9
“Hence, some suggestions were reported to partially solve this issue such as publishing a simulated dataset [105], providing complementary empirical results on an open-source benchmark dataset [106], or sharing model prediction and data labels to allow further statistical analysis [107].”
BackgroundColes, N. A., Perz, B., Behnke, M., Eichstaedt, J. C., Kim, S.-H., Vu, T. N., Raman, C., Tejada, J., Huynh, V. T., Zhang, G., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.241778 · Cited by 8
“Finally, requiring that teams make their materials and code openly available allowed us (and future researchers) to further inspect teams’ models, reproduce their solutions and identify approaches that seem most promising for follow-up research [62].”
Chen, D. C., Arnold, K., Sukhdeo, R., Alla, J. F., & Raman, S. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000733 · Cited by 16
Khairil, L., Benny, K. H. S., Jerry, J., Khatib, F. M., Ramli, M. D. C., & Kumar, S. (2026) · Pharmaceuticals · doi:10.3390/ph19060916 · Cited by 11
A Model-Driven Approach for Systematic Reproducibility and Replicability of Data Science Projects
Melchor, F., Rodríguez-Echeverría, R., Conejero, J. M., Prieto, Á. E., & Gutiérrez, J. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-07472-1_9 · Cited by 8
What is the vibration of effects?
Vinatier, C., Hoffmann, S., Patel, C. J., DeVito, N., Cristea, I. A., Tierney, B. T., Ioannidis, J. P. A., & Naudet, F. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112747 · Cited by 10
“…deep data), especially when data are not collected for research purpose 10 and need to be heavily preprocessed, 18 and (2) the complexity 19 of the models being considered (eg, linear models vs more complex ones), 20 as there are many more junctures where choices can be made with complex models.”
Self-citationClift, A. K., Hippisley‐Cox, J., Dodwell, D. J., Lord, S., Brady, M., Petrou, S., & Collins, P. G. S. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-050828 · Cited by 8
Ranftl, S., Rolf‐Pissarczyk, M., Wolkerstorfer, G., Pepe, A., Egger, J., von der Linden, W., & Holzapfel, G. A. (2022) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2022.115594 · Cited by 9
“Although deep CNNs have shown high accuracy on a large number of tasks, a limitation stems from their low robustness and reproducibility [89, 90], also given their intrinsic inability to express uncertainty [41].”
BackgroundTowards Better User Studies in Computer Graphics and Vision
Bylinskii, Z., Hertzmann, A., Zhang, Y., Herman, L., & Hutka, S. (2023) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000106 · Cited by 10
“, 2020; Peng, 2011), specifically within machine learning (Pineau et al., 2021; Haibe-Kains et al., 2020) and HCI (Greenberg and Thimbleby, 1991; Greiffenhagen and Reeves, 2014; Hornbæk et al.”
BackgroundKim, J. P. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.02.021 · Cited by 8
Continental generalization of an AI system for clinical seizure recognitionPreprint
Yang, Y., Truong, N. D., Maher, C., Nikpour, A., & Kavehei, O. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.03.07.433990 · Cited by 8
Automatic transparency evaluation for open knowledge extraction systems
Basereh, M., Caputo, A., & Brennan, R. (2023) · Journal of Biomedical Semantics · doi:10.1186/s13326-023-00293-9 · Cited by 10
“•Codes [49, 67] Table 11 Mean transparency results separated by transparency dimensions”
BackgroundRadiomic-based approaches in the multi-metastatic setting: a quantitative review
Geady, C., Patel, H., Peoples, J., Simpson, A. L., & Haibe‐Kains, B. (2025) · BMC Cancer · doi:10.1186/s12885-025-13850-5 · Cited by 7
Self-citationA Machine Learning Toolkit for Selecting Studies and Topics in Systematic Literature Reviews
Simonetti, A., Tumminello, M., Picone, P. M., & Minà, A. (2025) · Organizational Research Methods · doi:10.1177/10944281251341571 · Cited by 7
“Advancements in ML and NLP have the potential to offer signi fi cant value in automating complex tasks and handling large volumes of text (Haibe-Kains et al., 2020).”
AI and Open Science: Implications and Library Practice Recommendations
Helregel, N. (2025) · Library trends · doi:10.1353/lib.2025.a961191 · Cited by 8
“…sufficient documentation, explaining that even if the full datasets could not be shared due to privacy concerns, much more information about the development of the model, including the hyperparameter definitions, model predictions, and data labels, should have been shared (Haibe-Kains et al. 2020).”
Distributed Bandit-Based Cooperative Coevolution for Large-Scale Multi-Objective Data Publishing
Ge, Y.-F., Wang, H., Bertino, E., Cao, J., Zhang, Y., & Zheng, Z. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3609875 · Cited by 7
“Haibe-Kains et al. [1] reviewed data transparency and reproducibility issues in AI methods in medicine.”
Yue, S., Bajuri, N. H., Wang, B., & Samara, H. (2026) · Journal of Management & Governance · doi:10.1007/s10997-026-09783-1 · Cited by 7
Self-citationRigor and reproducibility instruction in academic medical libraries
LaPolla, F. W. Z., Bakker, C., Exner, N., Mentnech, T., Surkis, A., & Ye, H. (2022) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2022.1443 · Cited by 7
MIDRC CRP10 AI interface—an integrated tool for exploring, testing and visualization of AI models
Gorre, N., Carranza, E., Fuhrman, J. D., Li, H., Madduri, R., Giger, M. L., & Naqa, I. E. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acb754 · Cited by 7
Chen, H., Cao, X., Wang, J., Huang, J., Wang, Z., Wei, J., Zhao, D., Guo, Y., Peng, Y., & Miao, L. (2024) · ACS ES&T Engineering · doi:10.1021/acsestengg.3c00518 · Cited by 8
Wirz, C. D., Sutter, C., Demuth, J. L., Mayer, K. J., Chapman, W. E., Cains, M. G., Radford, J. T., Przybylo, V. M., Evans, A., Martin, T., et al. (2024) · Earth and Space Science · doi:10.1029/2023ea003364 · Cited by 7
Gonçalves-Pereira, J., Fernandes, J., Mendes, T. F., Gonzalez, F. A., & Fernandes, S. M. (2024) · Antibiotics · doi:10.3390/antibiotics13090853 · Cited by 7
“Another model is the whole-body physiologically based pharmacokinetic model, which compares the killing of bacteria with different antibiotic susceptibility [147].”
BackgroundLee, M. C. M., Farahvash, A., & Zezos, P. (2025) · Inflammatory Bowel Diseases · doi:10.1093/ibd/izaf050 · Cited by 9
Steele, L., Velazquez-Pimentel, D., & Thomas, B. (2021) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.17517 · Cited by 7
Afnan, M., Afnan, M. A. M., Liu, Y., Savulescu, J., Mishra, A., Conitzer, V., & Rudin, C. D. (2022) · Reproductive BioMedicine Online · doi:10.1016/j.rbmo.2022.03.015 · Cited by 7
Lin, Y., & Xiao, N. (2022) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221096117 · Cited by 7
Opportunities for integration of artificial intelligence into stereotactic radiosurgery practice
Kotecha, R. R., & Aneja, S. (2021) · Neuro-Oncology · doi:10.1093/neuonc/noab169 · Cited by 7
Integrating statistical design and inference: A roadmap for robust and trustworthy medical AI
Wei, Q., Cui, M., Liu, Z., Liu, Z., Feng, G., Li, Y., Christiani, D. C., Li, L., Wang, J., Hao, Y., et al. (2025) · The Innovation Medicine · doi:10.59717/j.xinn-med.2025.100145 · Cited by 6
The Importance of Ethical Reasoning in Next Generation Tech Education
Reis, S. S., Coelho, L., Sarmet, M., Araújo, J., & Corchado, J. M. (2023) · International Conference of the Portuguese Society for Engineering Education (CISPEE) · doi:10.1109/cispee58593.2023.10227651 · Cited by 7
“The problem is that users are often not aware of how these algorithms work, by whom they are used, what they are used for, or how they guarantee our confidentiality, which any user has the right to see clarified [45].”
BackgroundEthics-Aware Application of Digital Technologies in the Construction Industry
Khodabakhshian, A. (2024) · SpringerBriefs in applied sciences and technology · doi:10.1007/978-3-031-52962-7_5 · Cited by 6
Using real-word data to evaluate the effects of broadening eligibility criteria in oncology trials
Sanz‐García, E., Haibe‐Kains, B., & Siu, L. L. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.05.012 · Cited by 6
Self-citationBongrand, P. (2022) · Current Issues in Molecular Biology · doi:10.3390/cimb44020035 · Cited by 6
“First, the complexity of trained networks caused them to be considered as black boxes, and attempts are currently done to decrease their opacity [54,55] or increase reliability [56,57].”
BackgroundDevelopment of Predictive Models for Survival among Women with Breast Cancer in Malaysia
Kadir, M. N. N. A., Yaacob, N. M., Yusof, S. N., Hadi, I. S. A., Musa, K. I., Isa, S. A. M., Bahtiar, B., Adam, F., Yahya, M. M., & Hairon, S. M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192215335 · Cited by 6
Secure Convolutional Neural Network-Based Internet-of-Healthcare Applications
Khriji, L., Bouaafia, S., Messaoud, S., Ammari, A. C., & Machhout, M. (2023) · IEEE Access · doi:10.1109/access.2023.3266586 · Cited by 8
“However, only model weights may well be given as a potential substitute for sharing patient information to protect patient confidentiality [38], [39], [40].”
BackgroundRosendo, D., Keahey, K., Costan, A., Simonin, M., Valduriez, P., & Antoniu, G. (2023) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3589806.3600032 · Cited by 6
Generative AI: Impactful Considerations to Responsible Data Practices in Business Execution
Osmëni, T., & Ali, M. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384863 · Cited by 8
“[58] discusses this problem in the medical field by suggesting the need for transparency at every stage of the GenAI development, use of the training data and building of the model.”
BackgroundMapping invasive iceplant extent in southern coastal California using high-resolution aerial imagery
García, C. G., Brun, J., & Halpern, B. S. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102559 · Cited by 7
Intelligent Learning Management Systems
Vaerenbergh, S. V., & Pérez-Suay, A. (2022) · Advances in higher education and professional development book series · doi:10.4018/978-1-7998-9247-2.ch009 · Cited by 6
The Use of Machine Learning in Regulatory Drug Safety Evaluation
Zhang, D., Song, J., Dharmarajan, S. H., Jung, T. H., Lee, H., Ma, Y., Zhang, R., & Levenson, M. S. (2022) · Statistics in Biopharmaceutical Research · doi:10.1080/19466315.2022.2108135 · Cited by 6
Irion, K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4147375 · Cited by 6
Chen, D. C., He, E., Pace, K., Chekay, M., & Raman, S. (2025) · The Oncologist · doi:10.1093/oncolo/oyaf112 · Cited by 5
Faria, N., Campelos, S., & Carvalho, V. H. (2025) · Information · doi:10.3390/info16090740 · Cited by 7
Zhang, T., Sun, Z., Li, Z., Ahmad, M. U., Islam, M. T., Yang, X., Yang, X., Wu, L., Xie, J., Feng, W., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06181-3 · Cited by 5
“The clinical integration of artificial intelligence (AI) in medicine faces significant challenges related to trans-parency and reproducibility [51].”
Bugingo, C., Raju, T. J., Moparthi, S., Singh, J., Harish, M. S., Karim, S., & Turatsinze, A. N. (2026) · Seeds · doi:10.3390/seeds5010015 · Cited by 5
“Capacity-building under IPPC and OECD programs emphasizes competency-based training and harmonized documentation so that diverse laboratories can meet the same quality bar and integrate with national and international certification systems [288].”
Norén, G. N., Meldau, E.-L., & Ellenius, J. (2026) · Drug Safety · doi:10.1007/s40264-026-01649-7 · Cited by 7
“It may be relevant to present descriptive statistics for positive and negative controls.”
Seyhan, A. A., & Carini, C. (2026) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2026.105461 · Cited by 7
Abass, S. A. E., Altamimi, A. S. A., Aldakhil, T., & Prajapati, P. (2026) · Critical Reviews in Analytical Chemistry · doi:10.1080/10408347.2026.2694610 · Cited by 6
A Genealogical Approach to Algorithmic Bias
Ziosi, M., Watson, D., & Floridi, L. (2024) · Minds and Machines · doi:10.1007/s11023-024-09672-2 · Cited by 10
“Additionally, it has been suggested that external validation of models by trusted third parties can ensure the reproducibility of results and surface biases (Haibe-Kains et al., 2020).”
Ganeshkumar, C., Basu, R. J., Yuvaraj, M., & David, A. (2024) · Journal of Industrial Integration and Management · doi:10.1142/s2424862224500015 · Cited by 5
Aggarwal, R. K., Verma, T., & Aggarwal, A. (2024) · Advances in logistics, operations, and management science book series · doi:10.4018/979-8-3693-4350-0.ch013 · Cited by 5
Testing the consistency of performance scores reported for binary classification problems
Fazekas, A., & Kovács, G. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111993 · Cited by 7
Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI
Moulton, R. H., McCully, G. A., & Hastings, J. D. (2024) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars61786.2024.10778911 · Cited by 5
“While this paper does not attempt to be the source of solutions for this problem, the following subsections point to related work that aims to address some of the specific challenged faced during this study.”
Elazzouni, D. H., Alsunbul, H., Almnea, R. A., Awadh, W., Alshehri, A. S., Alhaidry, H., Javed, R., & Khan, A. A. (2026) · REVIEWS ON ADVANCED MATERIALS SCIENCE · doi:10.1515/rams-2025-0205 · Cited by 4
Li, X., Gao, X., Wang, C., Wang, Q., Li, B., & Wan, K. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3390426 · Cited by 4
“CurrentresearchonDLinterpretabilitymainlyfocuseson transparency[27],[28]andpost-hocinterpretability[29],[30].”
Meta-Repository of Screening Mammography Classifiers
Chłędowski, J., Stadnick, B., Witowski, J. S., Rajiv, V., Shamout, F. E., Cho, K., & Geras, K. J. (2025) · Studies in health technology and informatics · doi:10.3233/shti251263 · Cited by 4
Artificial Intelligence in Otolaryngology
Tai, K., Zhao, R., & Rameau, A. (2024) · Otolaryngologic Clinics of North America · doi:10.1016/j.otc.2024.04.008 · Cited by 4
Watanabe, A. T., Vu, H., Chim, C. Y., Litt, A. W., Retson, T., & Mayo, R. C. (2024) · Journal of Breast Imaging · doi:10.1093/jbi/wbae056 · Cited by 4
Metadata
- Authors
- Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., Mason, C. E., Jones, W., Dopazo, J., Furlanello, C., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C. S., Broderick, T., Hoffman, M. M., Leek, J. T., Korthauer, K., Huber, W., Brazma, A., Pineau, J., Tibshirani, R., Hastie, T., Ioannidis, J. P. A., Quackenbush, J., & Aerts, H. J. W. L.
- Affiliations
- University of Toronto (CA)
- Brigham and Women's Hospital (US)
- University Health Network (CA)
- National Center for Toxicological Research (US)
- University of Oxford (GB)
- SAS Institute (United States) (US)
- Q2 Solutions (United States) (US)
- City University of New York (US)
- Canadian Institute for Advanced Research (CA)
- Alex's Lemonade Stand Foundation (US)
- University of British Columbia (CA)
- European Bioinformatics Institute (GB)
- Stanford Medicine (US)
- ORCID
- Haibe-Kains, Benjamin
- Adam, George Alexandru
- Hosny, Ahmed
- Khodakarami, Farnoosh
- Kusko, Rebecca
- Sansone, Susanna-Assunta
- Tong, Weida
- Mason, Christopher E.
- Jones, Wendell
- Furlanello, Cesare
- Waldron, Levi
- Wang, Bo
- McIntosh, Chris
- Goldenberg, Anna
- Kundaje, Anshul
- Greene, Casey S.
- Broderick, Tamara
- Hoffman, Michael M.
- Leek, Jeffrey T.
- Korthauer, Keegan
- Huber, Wolfgang
- Brazma, Alvis
- Pineau, Joelle
- Tibshirani, Robert
- Hastie, Trevor
- Ioannidis, John P. A.
- Quackenbush, John
- Aerts, Hugo J. W. L.
- Venue
- Nature
- Published
- 2020-10-14
- Volume
- 586
- Issue
- 7829
- Pages
- E14-E16
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC8144864
- OpenAlex
- W3008058947
- Cited by
- 540
- References
- 18
- Citations by year
- 2026: 83 · 2025: 91 · 2024: 97 · 2023: 110 · 2022: 79 · 2021: 70 · 2020: 10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RISalud Institutional Health Repository of Andalucía (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Biotechnology and Biological Sciences Research Council (BB/L005069/2, BB/E025080/1, BB/L005069/1, BB/J020265/1, BB/L005050/1)
- National Cancer Institute (U24CA180996, R01CA237170, CA237170)
- National Institute of General Medical Sciences (DP2GM123485)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- reproducibility, explainability, deep learning, backtesting, generalizability, machine learning in healthcare, intellectual and developmental disabilities
- MeSH terms
- Algorithms, Artificial Intelligence, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
