Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 801–900 of 1,006
Potie, N., Giannoukakos, S., Hackenberg, M., & Fernández, A. (2019) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2019.8858976 · Cited by 15
“Throughout this work contribution we have stressed the relevance of EFS in the context of the emerging field of XAI [6], [29].”
BackgroundInfluentialJudging One’s Own or Another Person’s Responsibility in Interactions With Automation
Douer, N., & Meyer, J. (2020) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720820940516 · Cited by 15
“AI algorithms increase these difficulties, as the internal processes can be opaque (“black box”) and occasionally produce peculiar counterintuitive results (Castelvecchi, 2016; Scharre, 2016).”
BackgroundVorras, A., & Mitrou, L. (2021) · Springer eBooks · doi:10.1007/978-3-030-69583-5_10 · Cited by 18
Yaghy, A., Lee, A., Keane, P. A., Keenan, T. D. L., Mendonça, L. S. M., Lee, C. S., Cairns, A. M., Carroll, J., Chen, H., Clark, J., et al. (2022) · Experimental Eye Research · doi:10.1016/j.exer.2022.109092 · Cited by 12
Boukobza, A., Burgun, A., Roudier, B., & Tsopra, R. (2022) · JMIR Medical Informatics · doi:10.2196/34306 · Cited by 12
Majstorović, J., Giffard‐Roisin, S., & Poli, P. (2022) · Geophysical Journal International · doi:10.1093/gji/ggac369 · Cited by 16
Effects of Fairness and Explanation on Trust in Ethical AI
Angerschmid, A., Theuermann, K., Holzinger, A., Chen, F., & Zhou, J. J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14463-9_4 · Cited by 19
de León, G., Fiorentini, N., Leandri, P., & Losa, M. (2023) · Remote Sensing · doi:10.3390/rs15112722 · Cited by 18
Interpretable machine learning model for imaging-based outcome prediction after cardiac arrest
Liu, C., Elmer, J., Arefan, D., Pease, M. W., & Wu, S. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109894 · Cited by 12
A Fast Inference and Type-Reduction Process for Constrained Interval Type-2 Fuzzy Systems
D'Alterio, P., Garibaldi, J. M., John, R. I., & Wagner, C. (2020) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.3018379 · Cited by 11
“In recent years, there has been a growing trend for more explainable and transparent AI systems that has led to the creation of the new research field of explainable artificial intelligence (XAI) [1], [2].”
BackgroundNeuro-Symbolic Artificial Intelligence: Application for Control the Quality of Product Labeling
Golovko, V. A., Kroshchanka, A., Kovalev, M., Taberko, V., & Ivaniuk, D. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-60447-9_6 · Cited by 17
PTEN and DNA Ploidy Status by Machine Learning in Prostate Cancer
Cyll, K., Kleppe, A., Kalsnes, J., Vlatkovic, L., Pradhan, M., Kildal, W., Tobin, K. A. R., Reine, T. M., Wæhre, H., Brennhovd, B., et al. (2021) · Cancers · doi:10.3390/cancers13174291 · Cited by 11
Feldmann, C., & Bajorath, J. (2022) · Biomolecules · doi:10.3390/biom12040557 · Cited by 11
“Since ML model decisions are typically difficult to rationalize, the identification of structural features that determine accurate predictions requires the application of approaches for explaining ML [21–25].”
BackgroundExplainable Model Fusion for Customer Journey Mapping
Okazaki, K., & Inoue, K. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.824197 · Cited by 16
“As research on the application of machine learning to decision-making progressed, it became clear that the output process for learning-based predictions could not be visualized (Castelvecchi, 2016)—this is the so-called “black-box” problem (Ribeiro et al., 2016).”
BackgroundIT vendors’ legitimation strategies and market share: The case of EMR systems
Paré, G., Marsan, J., Jaana, M., Tamim, H., & Lukyanenko, R. (2020) · Information & Management · doi:10.1016/j.im.2020.103291 · Cited by 17
“In contrast, AI, especially the more advanced AI methods based on complex models (e.g., deep learning neural networks), are often dubbed “ black boxes ” because of their lack of transparency and explainability, often even to data scientists themselves [52 – 54].”
BackgroundInput Validation for Neural Networks via Runtime Local Robustness VerificationPreprint
Liu, J., Chen, L., Miné, A., & Wang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03339 · Cited by 17
Elyoseph, Z., Gur, T., Haber, Y., Simon, T., Angert, T., Navon, Y., Tal, A., & Asman, O. (2024) · doi:10.2196/preprints.58011 · Cited by 11
A Paradigm for Democratizing Artificial Intelligence Research
Moreau, E., Vogel, C., & Barry, M. (2019) · Intelligent Systems Reference Library · doi:10.1007/978-3-030-15939-9_8 · Cited by 10
“9 Of course, not every software prototype is worth being re-used, but out of the thousands of experimental papers published every year, very few prototypes get enough scientific and/or technological interest to be deemed worthy of devoting resources to their maintenance, development and…”
BackgroundAI-driven design and applications of quantum dots
Soni, P. K., Satnami, M. L., Nagwanshi, R., Chawre, Y., Kujur, A. B., Sinha, A., Miri, P., Karbhal, I., & Ghosh, K. K. (2026) · Nano Today · doi:10.1016/j.nantod.2026.102974 · Cited by 17
Lin, C., Chen, C.-C., Chau, T., Lin, C.-S., Tsai, S.-H., Lee, D.-J., Lee, C.-C., Shang, H.-S., & Lin, S.-H. (2022) · Clinica Chimica Acta · doi:10.1016/j.cca.2022.09.021 · Cited by 10
Unlocking the Black Box: Towards Interactive Explainable Automated Machine Learning
Garouani, M., & Bouneffa, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48232-8_42 · Cited by 11
Yang, M., Liu, S., Hao, T., Ma, C., Chen, H., Li, Y., Wu, C., Xie, J., Qiu, H., Li, J., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102785 · Cited by 10
Resilient Machine Learning: Advancement, Barriers, and Opportunities in the Nuclear Industry
Khadka, A., Sthapit, S., Epiphaniou, G., & Maple, C. R. (2024) · ACM Computing Surveys · doi:10.1145/3648608 · Cited by 14
“— Due to the lack of transparency and explainability of the ML models, there is uncertainty around the predictability and reliability of the output [26].”
BackgroundWang, F., Song, Y., Xu, H. Y., Liu, J., Tang, F., Yang, D., Yang, D., Liang, W., Ren, L., Wang, J., et al. (2024) · Photodiagnosis and Photodynamic Therapy · doi:10.1016/j.pdpdt.2024.104236 · Cited by 12
Ilich, N., & Todorović, A. (2024) · Hydrological Sciences Journal · doi:10.1080/02626667.2024.2394640 · Cited by 10
“This is all happening in spite of many reports that practitioners are having difficulties understanding the results and finding meaningful ways to apply them in practice (Castelvecci 2016, Quinn et al .”
Park, J., Seong, S., Park, S., Kim, M., & Kim, H. Y. (2024) · Journal of Safety Research · doi:10.1016/j.jsr.2024.10.002 · Cited by 15
AI in Scientific Research: Empowering Researchers with Intelligent Tools
-, S. P., Kalva, P., & Kommidi, V. R. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit241051012 · Cited by 21
Improved Prediction of Ligand–Protein Binding Affinities by Meta-modeling
Lee, H., Emani, P. S., & Gerstein, M. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01116 · Cited by 14
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 22
“Linked to trust is the ‘black box’ problem—a problem particular to AI, where users cannot inspect and understand the inner workings of a system [29].”
“Turning right”? An experimental study on the political value shift in large language models
Liu, Y., Panwang, Y., & Gu, C. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04465-z · Cited by 31
“(Castelvecchi 2016).”
Conformal Depression Prediction
Li, Y., Qu, S., & Zhou, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3542023 · Cited by 18
“While the rapid development of deep learning and computer vision has led to exciting progress in depression prediction, the reliability and trustworthiness of deep learning models-considered black boxes [23], [24], [25]-have not been well addressed.”
Yu, B., Xing, H., Ge, W., Yan, J., & Li, Y.-A. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-01915-9 · Cited by 16
“Due to their decision-making process and logic often being considered a “black box” (Castelvecchi 2016; Rudin 2019) these models may not always gain acceptance from decision-makers.”
Shi, W., Wan, X., Zhao, F., & Deng, R. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106570 · Cited by 10
Konnai, Y., Fukumoto, K., Takeuchi, M., Takeuchi, R., Fujiwara, S., Yasumizu, Y., Tanaka, N., Takeda, T., Matsumoto, K., Kosaka, T., et al. (2025) · Annals of Surgical Oncology · doi:10.1245/s10434-025-18590-1 · Cited by 16
“32 It is challenging for us to understand why the AI made a particular decision or why it made an incorrect decision.”
Fagerholm, E. D., Dinov, M., Knöpfel, T., & Leech, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197893 · Cited by 11
Deep Learning for Autonomous Vehicle Control: Algorithms, State-of-the-Art, and Future Prospects
Kuutti, S., Fallah, S., Bowden, R., & Barber, P. (2019) · Synthesis lectures on advances in automotive technology · doi:10.2200/s00932ed1v01y201906aat008 · Cited by 33
Bias and Discrimination in Artificial Intelligence: Emergence and Impact in E-Business
Weyerer, J. C., & Langer, P. F. (2020) · Advances in e-business research series · doi:10.4018/978-1-7998-1879-3.ch011 · Cited by 14
AI applications in prevalent diseases and disorders
Catania, L. J. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-824477-7.00007-9 · Cited by 12
A Comprehensive Study of Applying Object Detection Methods for Medical Image Analysis
Ganatra, N. (2021) · International Conference on Computing for Sustainable Global Development · doi:10.1109/indiacom51348.2021.00147 · Cited by 19
“This makes it difficult to understand and often termed as black box problem solving approach [49].”
BackgroundAŞKUN, V. (2023) · Erciyes Akademi · doi:10.48070/erciyesakademi.1281544 · Cited by 15
“ChatGPT gibi derin öğrenme modellerinin karmaşıklığı ve bulanıklığı, araştırmacıların tahminlerini veya metin üretimini yönlendiren temel mekanizmaları anlamalarını zorlaştırabilir (Castelvecchi, 2016) .”
Chhina, A., Trehan, K., Saini, M., Thakur, S., Kaur, M., Shahtaghi, N. R., Shivgotra, R., Soni, B., Modi, A., Bakrey, H., et al. (2023) · Current Pharmaceutical Design · doi:10.2174/1381612829666230807161421 · Cited by 13
Millarch, A. S., Bonde, A., Bonde, M., Klein, K. V., Folke, F., Rudolph, S. S., & Sillesen, M. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1249258 · Cited by 10
“Particularly neural networks have generally received criticism for “ black boxing ” how the model acts internally (36).”
BackgroundBack to the Feature: A Neural-Symbolic Perspective on Explainable AI
Campagner, A., & Cabitza, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_3 · Cited by 12
“…the model learned by a DL architecture is often imper-scrutable (due to the complex interactions between the high number of non-linear functions) [13] and possibly subject to a variety of attacks [28]: both problems (also in combination) could hinder the actual applicability of these systems in…”
Machine Learning for Supplementing Behavioral Assessment
Bailey, J. D., Baker, J. C., Rzeszutek, M. J., & Lanovaz, M. J. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-020-00273-9 · Cited by 15
Kim, H. H., Kim, Y., & Park, Y. R. (2021) · JMIR mhealth and uhealth · doi:10.2196/22183 · Cited by 11
Alamin, M. A. A., & Uddin, G. (2021) · IEEE International Conference on Autonomous Systems (ICAS) · doi:10.1109/icas49788.2021.9551151 · Cited by 22
Black Box Models for eXplainable Artificial Intelligence
Chennam, K. K., Mudrakola, S., Maheswari, V. U., Aluvalu, R., & Rao, K. G. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-12807-3_1 · Cited by 13
Gong, K., Dai, Q., Wang, J., Zheng, Y., Shi, T., Yu, J., Chen, J., Huang, S., & Wang, Z. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1118340 · Cited by 10
“This is called the “black box” problem of deep neural networks (Castelvecchi, 2016).”
BackgroundWidmer, C. L., Sarker, M. K., Nadella, S., Fiechter, J., Juvina, I., Minnery, B., Hitzler, P., Schwartz, J., & Raymer, M. (2023) · Journal of Web Semantics · doi:10.1016/j.websem.2023.100807 · Cited by 10
Influencing laughter with AI-mediated communication
Mills, G. J., Gregoromichelaki, E., Howes, C., & Maraev, V. (2021) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.00011.mil · Cited by 10
The Augmented Qualitative Researcher: Using Generative AI in Qualitative Text AnalysisPreprint
Ibrahim, E. I., & Voyer, A. M. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/gkc8w · Cited by 21
Demystifying the black box: an overview of explainability methods in machine learning
Kinger, S., & Kulkarni, V. Y. (2023) · International Journal of Computers and Applications · doi:10.1080/1206212x.2023.2285533 · Cited by 12
Enhancing AI-CDSS with U-AnoGAN: Tackling data imbalance
Mun, C., Ha, H., Lee, O., & Cheon, M. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107954 · Cited by 13
The utilization of artificial intelligence in glaucoma: diagnosis versus screening
AlShawabkeh, M., AlRyalat, S. A., Bdour, M. D. A., Alni’mat, A., & Al-Akhras, M. (2024) · Frontiers in Ophthalmology · doi:10.3389/fopht.2024.1368081 · Cited by 16
Zapaishchykova, A., Tak, D., Ye, Z., Liu, K. X., Likitlersuang, J., Vajapeyam, S., Chopra, R., Seidlitz, J., Bethlehem, R. A. I., Mak, R. H., et al. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00114 · Cited by 12
“Interpretability of deep learning algorithms is clouded by the black-box nature of hidden layers ( Castelvecchi, 2016), and brain age models to date have not investigated the underlying biological and anatomical bases of predictions.”
BackgroundRoth, J. P., & Bajorath, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-57135-6 · Cited by 12
Cui, W., Chen, K., Wei, G., Lyu, M., & Zhu, F. (2024) · Geophysical Journal International · doi:10.1093/gji/ggae140 · Cited by 9
Statistical modeling of equilibrium phase transition in confined fluids
Auti, G., Paul, S., Hsu, W., Shohei Chiashi, A., Maruyama, S., & Daiguji, H. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.023333 · Cited by 11
“However, even the most current deep-learning models are black boxes [44, 45], meaning that they perform the majority of their calculations internally, so that significant thermodynamic information is overlooked.”
Remote Sensing Technology for Observing Tree Mortality and Its Influences on Carbon–Water Dynamics
Ni, M., Wu, Q., Li, G., & Li, D. (2025) · Forests · doi:10.3390/f16020194 · Cited by 15
“…undermines confidence in the practical application of neural network models for image recognition and classification, posing ongoing challenges [140], particularly when addressing issues such as spectral variabilities caused by various degradation mechanisms in hyperspectral remote sensing,…”
Development and validation of the trust in AI scale (TAIS)
Wischnewski, M., Doebler, P., & Krämer, N. C. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2715475 · Cited by 19
Artificial intelligence and human decision making: Exploring similarities in cognitive bias
Campbell, H., Goldman, S., & Markey, P. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100138 · Cited by 23
Turgut, D., & Künüroğlu, F. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2474474 · Cited by 13
“Unlike tangible technologies like computers or robots, AI remains an abstract algorithmic entity, often perceived by users as a “black box,” making its full potential and risks difficult to grasp (Castelvecchi, 2016; Tan et al., 2009).”
Liang, L., Sun, Y., & Yang, B. (2025) · Aslib Journal of Information Management · doi:10.1108/ajim-08-2024-0684 · Cited by 11
Zhao, H., Miao, C., Zhu, Y., Shu, Y., Wu, X., Yin, Z., Deng, X., Gong, W., Yang, Z., & Zou, W. (2025) · BMC Cancer · doi:10.1186/s12885-025-14462-9 · Cited by 14
“This is the reason why deep learning has long been criticized as a “black box” [19].”
Bertram, C., Schutten, M. M., Ressel, L., Breininger, K., Webster, J. D., & Aubreville, M. (2025) · Veterinary Pathology-Supplement · doi:10.1177/03009858251344320 · Cited by 11
“1,6 In addition, depending on how the model is applied (computer-assisted approach), the interplay of experts with those models carries the risk of a multitude of cognitive biases.”
Curran, N. M., Gu, B., Zhen, L., & Jenks, C. J. (2025) · RELC Journal · doi:10.1177/00336882251367459 · Cited by 16
Rethinking Responsibility in the Digital Age: A Narrative Approach
de Vaujany, F., Leclercq‐Vandelannoitte, A., Aroles, J., Introna, L. D., & Davidson, S. (2025) · MIS Quarterly · doi:10.25300/misq/2025/17970 · Cited by 20
Qualitative research with LLM chatbots: Technological reflexivity for interpretative technology
Ibrahim, E. I., & Voyer, A. M. (2025) · Qualitative Research · doi:10.1177/14687941251390794 · Cited by 18
“Skepticism toward AI often concerns what is referred to as the “ black box ” problem (Castelvecchi, 2016).”
Ji, A., Zhu, Y., Fan, H., Xue, X., Zhang, M., & Zhou, J. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111948 · Cited by 13
Ding, X., & Xing, C. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105945 · Cited by 11
Langer, P. F., & Weyerer, J. C. (2020) · Springer eBooks · doi:10.1007/978-3-658-30997-8_11 · Cited by 15
On the reproducibility of enzyme reactions and kinetic modelling
Gygli, G. (2022) · Biological Chemistry · doi:10.1515/hsz-2021-0393 · Cited by 9
“This can lead to limitations in the application of these algorithms (Castelvecchi, 2016; Doshi-Velez and Kim, 2017; Poon and Sung, 2021).”
MethodNonlinear Hyperparameter Optimization of a Neural Network in Image Processing for Micromachines
Shen, M., Yang, J., Li, S., Zhang, A., & Bai, Q. (2021) · Micromachines · doi:10.3390/mi12121504 · Cited by 10
Blaha, L. M., Abrams, M., Bibyk, S., Bonial, C., Hartzler, B. M., Hsu, C. D., Khemlani, S., King, J. M., Amant, R. S., Trafton, J. G., et al. (2022) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2022.800280 · Cited by 24
Unfreezing autonomous vehicles with game theory, proxemics, and trust
Camara, F., & Fox, C. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.969194 · Cited by 9
“The deep learning models reviewed—which are needed for low-level understanding of pedestrians and used as input to the higher level interaction—-may be legally and practically problematic in the event of accidents and their subsequent investigations, as their black-box nature usually precludes human understanding and…”
BackgroundHozumi, H., & Shimizu, H. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad133 · Cited by 11
“Moreover, ML methods, including NNs, often lack transparency due to the complexity arising from neural connections and mathematical abstractions (16–18), making it potentially impossible to explain their predictions.”
MethodRole of ChatGPT and Google Bard in the Diagnosis of Psychiatric Disorders: A Comparative Study
Abbas, Y. N., Mahmood, Y. M., Hassan, H. A., Hamad, D. Q., Hasan, S. J., Omar, D. A., Kakamad, S. H., Hamasalih, H. M., Hassan, M. N., Rahim, H. M.-A., et al. (2023) · Barw Medical Journal · doi:10.58742/4vd6h741 · Cited by 15
“DSM-5 post-traumatic stress disorder (PTSD). often termed the 'black box' phenomenon; the exact rationale behind an AI's diagnosis is not always transparent, which can be a hindrance in clinical settings where understanding the reasoning is pivotal [9].”
BackgroundCrespo-Otero, A., Esteve, P., & Zanin, M. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.115359 · Cited by 10
Kuteynikov, D. L., Аликович, И. О., Lebedev, V., & Zenin, S. (2019) · Lex Russica · doi:10.17803/1729-5920.2019.154.9.085-095 · Cited by 9
TEMPORAL REPRESENTATION OF CAUSALITY IN THE CONSTRUCTION OF EXPLANATIONS IN INTELLIGENT SYSTEMS
Chalyi, S. F., & Leshchynskyi, V. O. (2020) · Advanced Information Systems · doi:10.20998/2522-9052.2020.3.16 · Cited by 9
Explainable Hopfield Neural Networks Using an Automatic Video-Generation System
Rubio-Manzano, C., Segura-Navarrete, A., Martínez-Araneda, C., & Vidal‐Castro, C. (2021) · Applied Sciences · doi:10.3390/app11135771 · Cited by 8
Ethical Challenges of Integrating AI into Healthcare
Lehmann, L. S. (2021) · Springer eBooks · doi:10.1007/978-3-030-58080-3_337-1 · Cited by 15
Privacy-Preserving Mechanisms with Explainability in Assistive AI Technologies
Müftüoğlu, Z., Kızrak, M. A., & Yıldırım, T. (2021) · Learning and analytics in intelligent systems · doi:10.1007/978-3-030-87132-1_13 · Cited by 19
Benchmarking feature selection methods for compressing image information in high-content screening
Siegismund, D., Fassler, M., Heyse, S., & Steigele, S. (2022) · SLAS TECHNOLOGY · doi:10.1016/j.slast.2021.10.015 · Cited by 14
On the Need for Collaborative Intelligence in Cybersecurity
Martin, T. (2022) · Electronics · doi:10.3390/electronics11132067 · Cited by 11
“The ubiquity of data-driven intelligent systems has led to a demand for so-called "explainability" [16].”
BackgroundMachines with vision for intraoperative guidance during gastrointestinal cancer surgery
Khalid, M. U., Laplante, S. J., & Madani, A. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.1025382 · Cited by 10
“, unexplainable) processes with which machines can learn information and subsequently make decisions in supervised, semi-supervised, and unsupervised settings (9).”
BackgroundZhang, J., Sami, S. A. K., & Meiser‐Stedman, R. (2022) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2022.102642 · Cited by 12
Comparison of Learning Spacecraft Path-Planning Solutions from Imitation in Three-Body Dynamics
Parmar, K., Taheri, E., & Guzzetti, D. (2023) · Journal of Spacecraft and Rockets · doi:10.2514/1.a35458 · Cited by 9
Bemthuis, R., Wang, W., Iacob, M., & Havinga, P. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.03.057 · Cited by 9
“…sophistication of data-driven artificial intelligence (AI) approaches has recently increased to such an extent that human intervention is minimized [6], we also need to understand what the underlying models are doing, instead of treating computational machines as merely black box models [16, 38].”
BackgroundGuevara, E., Pierre, W. C., Tessier, C., Akakpo, L., Londoño, I., Lesage, F., & Lodygensky, G. A. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00358 · Cited by 12
“One limitation of ANN, and machine learning in general, is the fact that the only observed states are the inputs and the outputs, making interpretation of their inner functions exceedingly difficult (Castelvecchi, 2016).”
BackgroundThe effect of variable labels on deep learning models trained to predict breast density
Squires, S., Harkness, E. F., Evans, D. G., & Astley, S. (2023) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/accaea · Cited by 8
Ni, D., & Cozzolino, D. (2023) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2023.117067 · Cited by 9
Machine Learning in Invasive and Noninvasive Coronary Angiography
Ünlü, O., & Fahed, A. C. (2023) · Current Atherosclerosis Reports · doi:10.1007/s11883-023-01178-z · Cited by 11
“Finally, a significant criticism of ML, especially deep learning models, is their “black box” nature [54].”
Choi, H., Cho, Y., Min, C., Kim, K., Kim, E., Lee, S., & Kim, J. (2024) · Digital Health · doi:10.1177/20552076241256730 · Cited by 13
“Addressing the challenge of model interpretability inherent in neural networks, 49,50 we employed IGs.”
MethodLiu, J., & Li, D. M. (2024) · Asian Journal of Criminology · doi:10.1007/s11417-024-09429-x · Cited by 21
“…we argue that the concept of interpretability, including its functions, explanation typologies, and the probability of achieving it, is ambiguous (Arrieta et al., 2020; Bathaee, 2017; Castelvecchi, 2016; Herm et al., 2023; Lipton, 2018; Marcinkevics & Vogt, 2020; Mohseni et al., 2021; Rudin, 2019).”
Kuo, C., Liu, G.-T., Lee, C.-E., Wu, J., Casimo, K., Weaver, K. E., Lo, Y., Chen, Y.-Y., Huang, W.-C., & Ojemann, J. G. (2024) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2024.110251 · Cited by 8
Hu, Z., Wang, Y., Ji, X., Xu, B., Li, Y., Zhang, J., Liu, X., Li, K., Zhang, J., Zhu, J., et al. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1413560 · Cited by 10
Jeantet, L., Zondo, K., Delvenne, C., Martin, J., Chevallier, D., & Dufourq, E. (2024) · Journal of Experimental Biology · doi:10.1242/jeb.249232 · Cited by 8
Li, L., Geng, Y., Chen, T., Lin, K., Xie, C., Qi, J., Wei, H., Wang, J., Wang, D., Yuan, Z., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-024-06017-6 · Cited by 10
“Although we revealed that the distinct image features in cancer surrounding tissue could be exploited to improve cancer diagnosis, it is still difficult to interpret how the deep learning model analyze the images and works as a "black box" [56].”
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.