Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 501–600 of 1,006
Automated end-to-end management of the modeling lifecycle in deep learning
Gharibi, G., Walunj, V., Nekadi, R., Marri, R., & Lee, Y. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09894-9 · Cited by 36
“Consequently, it is not uncommon to refer to DNNs as black box models/functions (Castelvecchi 2016; Vartak and Madden 2018b; Garcia et al. 2018).”
BackgroundPrivacy-Preserving and Explainable AI in Industrial Applications
Ogrezeanu, I. A., Vizitiu, A., Ciusdel, C. F., Puiu, A., Coman, S., Boldișor, C., Itu, A., Demeter, R., Moldoveanu, F. D., Suciu, C., et al. (2022) · Applied Sciences · doi:10.3390/app12136395 · Cited by 40
“A representative example was described in [45]; a robotics graduate student tried in 1991 to train a military vehicle to self-drive.”
BackgroundBrain Morphometry Estimation: From Hours to Seconds Using Deep Learning
Rebsamen, M., Suter, Y., Wiest, R., Reyes, M., & Rummel, C. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.00244 · Cited by 29
“Despiteprogress to improve the interpretability of deep learning (93), deep neural networks are still considered, to a large extent, as black boxes (94).”
BackgroundTancev, G., & Pascale, C. (2020) · Sensors · doi:10.3390/s20216198 · Cited by 28
Feldmann, C., Philipps, M., & Bajorath, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-01099-4 · Cited by 30
Legal framework for the coexistence of humans and conscious AI
Kiškis, M. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1205465 · Cited by 46
“Human comprehension of AI is not exhaustive by any means (LeCun et al., 2015; Castelvecchi, 2016).”
BackgroundMekheimer, M. A. A. G. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13750-y · Cited by 83
“…(AI), particularly generative AI tools like ChatGPT, presents both unprecedented opportunities and significant challenges for education (Baidoo-Anu & Ansah, 2023 ; Castelvecchi, 2016 ; Luo et al., 2024; Chen et al., 2025 ; Wohlfart & Wagner, 2025 ; Xu et al., 2024 ; Lan et al., 2024 ; Abedi, 2023).”
Dealing with Robustness of Convolutional Neural Networks for Image Classification
Arcaini, P., Bombarda, A., Bonfanti, S., & Gargantini, A. (2020) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest49225.2020.00009 · Cited by 34
“This is due to the fact that machine learning is almost a “black box” technique [6].”
BackgroundDeep autoregressive models with spectral attention
Moreno-Pino, F., Olmos, P. M., & Artés-Rodríguez, A. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109014 · Cited by 35
Filtjens, B., Ginis, P., Nieuwboer, A., Afzal, M. R., Spildooren, J., Vanrumste, B., & Slaets, P. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01699-0 · Cited by 29
“there is a lack of direct understanding of the models’ ability to identify relevant features [42].”
BackgroundGashi, M., Vuković, M., Jekic, N., Thalmann, S., Holzinger, A., Jean-Quartier, C., & Jeanquartier, F. (2022) · BioMedInformatics · doi:10.3390/biomedinformatics2010009 · Cited by 30
“In addition, xAI can also increase user acceptance, and the application rate of these models [12].”
BackgroundStorey, V. C., Lukyanenko, R., Maaß, W., & Parsons, J. (2022) · Communications of the ACM · doi:10.1145/3490699 · Cited by 40
Wang, J., Wang, Z., Chen, M., Xiao, Y., Chen, S., Wu, L., Yao, L., Jiang, X., Li, J., Xu, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00729-z · Cited by 28
“A growing number of people believe that the “ black box ” features of AI attenuated its reliability 20 .”
Lolak, S., Attia, J., McKay, G. J., & Thakkinstian, A. (2023) · JMIR Cardio · doi:10.2196/47736 · Cited by 31
Artificial intelligence in skeletal metastasis imaging
Dong, X., Chen, G., Zhu, Y., Ma, B., Ban, X., Wu, N., & Ming, Y. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.007 · Cited by 24
“Moreover, the black-box nature of deep learning algorithms has much controversy [60].”
BackgroundQin, X., Hu, X., Xiao, W., Zhu, C., Ma, Q., & Zhang, C. (2023) · Journal of Hepatocellular Carcinoma · doi:10.2147/jhc.s400166 · Cited by 25
“An unexplained neural network system and its application in medical imaging are typically called “black boxes”.(32) We Table 2 Performance of the Three Models”
Torres‐Martos, Á., Anguita‐Ruiz, A., Bustos‐Aibar, M., Ramírez‐Mena, A., Arteaga, M., Bueno, G., Leis, R., Aguilera, C. M., Alcalá, R., & Alcalá‐Fdez, J. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102962 · Cited by 32
Walker, J. M., Coraddu, A., & Oneto, L. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.119263 · Cited by 31
“…to the many different approximations that stem from Problem (2) to Problem (7) (i.e., due to parametrization in Step (1), surrogation in Step (2) (Castelvecchi, 2016; D’Amato et al., 2022; Kalikatzarakis et al., 2023; Coraddu et al., 2023), and optimization in Step (3) (Huang and Yang, 2016;…”
BackgroundMethodInfluentialAppraising Regulatory Framework Towards Artificial General Intelligence (AGI) Under Digital Humanism
Cheng, L., & Gong, X. (2024) · International Journal of Digital Law and Governance · doi:10.1515/ijdlg-2024-0015 · Cited by 34
AI in digital sports coaching – a systematic review
Jud, M., & Thalmann, S. (2025) · Managing Sport and Leisure · doi:10.1080/23750472.2024.2449016 · Cited by 32
“Modern AI is often associated with the system’s capacity to learn from data and perform tasks and business functions that traditionally require human cognition (Castelvecchi, 2016).”
The integration of artificial intelligence in assisted reproduction: a comprehensive review
Kakkar, P., Gupta, S., Paschopoulou, K. I., Paschopoulos, I., Paschopoulos, I., Siafaka, V., & Tsonis, O. (2025) · Frontiers in Reproductive Health · doi:10.3389/frph.2025.1520919 · Cited by 35
“This collaborative approach ensures a thoughtful examination of the quantity and quality of data used, emphasizing the necessity for transparency before the widespread implementation of AI in clinical practice (43).”
Improta, G., Ricciardi, C., Cesarelli, G., D’Addio, G., Bifulco, P., & Cesarelli, M. (2020) · Health and Technology · doi:10.1007/s12553-020-00449-y · Cited by 24
Przedborski, M., Smalley, M., Thiyagarajan, S., Goldman, A. J., & Kohandel, M. (2021) · Communications Biology · doi:10.1038/s42003-021-02393-7 · Cited by 29
Nazari, M., Kluge, A., Apostolova, I. I., Klutmann, S., Kimiaei, S., Schroeder, M., & Buchert, R. (2021) · Scientific Reports · doi:10.1038/s41598-021-02385-x · Cited by 23
Zakharova, I. G., Vorobeva, M. S., & Boganyuk, Y. V. (2022) · The Education and science journal · doi:10.17853/1994-5639-2022-1-163-190 · Cited by 43
Du, L., Yuan, J., Gan, M., Li, Z., Wang, P., Hou, Z., & Wang, C. (2022) · BMC Medical Imaging · doi:10.1186/s12880-022-00946-8 · Cited by 28
“Another major problem associated with deep learning is the issue of black box, as the complicated hierarchical representation of training data makes the prediction very difficult to understand and interpret [36].”
BackgroundComparison of three machine learning methods to estimate myocardial stiffness
Mehdi, R. R., Mendiola, E. A., Sears, A., Choudhary, G., Jacques Ohayon, A., Pettigrew, R. I., & Avazmohammadi, R. (2023) · Elsevier eBooks · doi:10.1016/b978-0-32-389967-3.00025-1 · Cited by 28
Lee, E.-M., & Kam, J. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.130177 · Cited by 24
In Praise of Artifice Reloaded: Caution With Natural Image Databases in Modeling Vision
Martínez-Garcia, M., Bertalmío, M., & Malo, J. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00008 · Cited by 30
“In fact, there is an active debate about the actual scientific gain of non-interpretable models, such as blind regression (Castelvecchi, 2016; Bohannon, 2017).”
BackgroundArtificial intelligence driven next-generation renal histomorphometry
Santo, B. A., Rosenberg, A. Z., & Sarder, P. (2020) · Current Opinion in Nephrology & Hypertension · doi:10.1097/mnh.0000000000000598 · Cited by 27
Nawrocki, T., Maldjian, P. D., Slasky, S. E., & Contractor, S. G. (2018) · Academic Radiology · doi:10.1016/j.acra.2017.12.027 · Cited by 23
Performance in the Workplace: a Critical Evaluation of Cognitive Enhancement
Acartürk, C., & Mücen, B. (2022) · NanoEthics · doi:10.1007/s11569-021-00407-6 · Cited by 28
“AI systems have been frequently criticized for being black box systems; they cannot respond to how they learn from data [26].”
BackgroundEffect of external loads on chloride ingress into concrete: A state-of-the-art review
Guo, B., Chu, J., Zhang, Z., Wang, Y., & Niu, D. (2024) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2024.138657 · Cited by 25
Towards eXplicitly eXplainable Artificial Intelligence
Kalmykov, V., & Kalmykov, L. V. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103352 · Cited by 30
“This is the black box problem of modern AI (Castelvecchi 2016).”
BackgroundTarget identification and assessment in the era of AI
Pun, F. W., Podolskiy, D. I., Izumchenko, E., Mortlock, A. A., Oprea, T. I., Scheibye‐Knudsen, M., Fortney, K., Morgen, E. K., Ren, F., & Zhavoronkov, A. A. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01412-8 · Cited by 40
“This opacity impedes the evaluation of these models by domain experts, affecting their acceptance 207 .”
Visualizing complex feature interactions and feature sharing in genomic deep neural networks
Liu, G., Zeng, H., & Gifford, D. K. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2957-4 · Cited by 28
A technique for determining relevance scores of process activities using graph-based neural networks
Stierle, M., Weinzierl, S., Harl, M. V., & Matzner, M. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113511 · Cited by 34
Socio-cognitive biases in folk AI ethics and risk discourse
Laakasuo, M., Herzon, V., Perander, S., Drosinou, M., Sundvall, J. R. I., Palomäki, J., & Visala, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00060-5 · Cited by 30
Software documentation is not enough! Requirements for the documentation of AI
Königstorfer, F., & Thalmann, S. (2021) · Digital Policy Regulation and Governance · doi:10.1108/dprg-03-2021-0047 · Cited by 35
Inland harmful algal blooms (HABs) modeling using internet of things (IoT) system and deep learning
Kwon, D. H., Hong, S. M., Abbas, A., Pyo, J., Lee, H., Baek, S., & Cho, K. H. (2022) · Environmental Engineering Research · doi:10.4491/eer.2021.280 · Cited by 27
“However, these models are limited to explain the correlation between 38 input and output variables, and the observation [27, 28].”
BackgroundApplications of Soft Computing in nuclear power plants: A review
Ramezani, I., Moshkbar-Bakhshayesh, K., Vosoughi, N., & Ghofrani, M. B. (2022) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2022.104253 · Cited by 22
Recognizing Semi-Natural and Spontaneous Speech Emotions Using Deep Neural Networks
Amjad, A., Khan, L., Ashraf, N., Mahmood, M. B., & Chang, H.-T. (2022) · IEEE Access · doi:10.1109/access.2022.3163712 · Cited by 28
“Although these methods have taken a signifi-cant step in image classification, it is still not a universal answer to the ‘‘black box’’ issue [98], [99].”
MethodSandoval, C. D. A., Cuate, O., González, L. C., Trujillo, L., & Schütze, O. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109103 · Cited by 23
“However, such an approach’s negative side-effect is the loss of model interpretability, given that the features are in a black-box form [55].”
BackgroundBae, S., Lee, H., Shin, J., Kim, H. S., Kim, Y., Kim, D. H., & Lee, J. M. (2022) · Chemistry of Materials · doi:10.1021/acs.chemmater.2c01092 · Cited by 24
Xu, X., Jia, Q., Yuan, H., Qiu, H., Dong, Y., Xie, W., Yao, Z., Zhang, J., Nie, Z., Li, X., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102953 · Cited by 27
Chen, X., Yu, Q., Peng, J., He, Z., Li, Q., Ning, Y., Gu, J., Lv, F., Jiang, H., & Xie, K. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.06.029 · Cited by 23
Xia, H., Tang, J., Qiao, J., Zhang, J., & Yu, W. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-06809-7 · Cited by 27
“Journals, such as ‘‘Nature,’’ ‘‘Science,’’ and ‘‘MIT Technology Review,’’ have declared that interpretability is a necessary condition for deep learning development [165].”
Artificial intelligence applied to MRI data to tackle key challenges in multiple sclerosis
Collorone, S., Coll, L., Lorenzi, M., Lladó, X., Sastre‐Garriga, J., Tintoré, M., Montalbán, X., Rovira, À., Pareto, D., & Tur, C. (2024) · Multiple Sclerosis Journal · doi:10.1177/13524585241249422 · Cited by 28
“49 The consequences of this are twofold: from a methodological perspective, this issue may inadvertently lead to error propagation in data analysis, since it may be virtually impossible to evaluate erroneous paths, causing the same model to obtain different results on different data sets.”
Incremental Residual Concept Bottleneck Models
Shang, C., Zhou, S., Zhang, H., Ni, X., Yang, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01049 · Cited by 78
“However, due to their complex and deep structures, they are often regarded as black-box models [7, 50], which are too difficult to understand and interpret.”
BackgroundFrontier AI developers need an internal audit function
Schuett, J. (2024) · Risk Analysis · doi:10.1111/risa.17665 · Cited by 54
Explainability and vision foundation models: A survey
Kazmierczak, R., Berthier, É., Frehse, G., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103184 · Cited by 61
A versatile XAI-based framework for efficient and explainable intrusion detection systems
Nugraha, B., Jnanashree, A. V., & Bauschert, T. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01118-9 · Cited by 41
Ortega, R. F., Irurita, J., Campo, E. J. E., & Mesejo, P. (2021) · International Journal of Legal Medicine · doi:10.1007/s00414-021-02660-6 · Cited by 28
“One of the main future challenges is to develop explainable AI methods, shifting from current opaque and non-intuitive models to others being able to provide an explanation to the user that justifies the output obtained [59].”
BackgroundDagli, M. M., Rajesh, A., Asaad, M., & Butler, C. E. (2021) · The American Surgeon · doi:10.1177/00031348211065101 · Cited by 26
“Although this is a limitation of AI, it is also a strength and explains why AI is unique since it reflects the actual real-life situations where variables interact in a very complex manner.(111,112)”
BackgroundDiagnosing Systemic Disorders with AI Algorithms Based on Ocular Images
Li, H., Cao, J., Grzybowski, A. E., Jin, K., Lou, L., & Ye, J. (2023) · Healthcare · doi:10.3390/healthcare11121739 · Cited by 29
Kanda, E., Epureanu, B. I., ADACHI, T., Tsuruta, Y., Kikuchi, K., Kashihara, N., Abe, M., Masakane, I., & Nitta, K. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233491 · Cited by 24
“, SVM and deep learning models, is a black box [21].”
BackgroundGood proctor or "Big Brother"? AI Ethics and Online Exam Supervision TechnologiesPreprint
Coghlan, S., Miller, T., & Paterson, J. (2020) · arXiv.org · doi:10.48550/arxiv.2011.07647 · Cited by 43
“Compared to, perhaps, the plagiarism detection tool Turnitin, proctoring AI may strike users as highly opaque [Castelvecchi, 2016].”
BackgroundBig Tech, Algorithmic Power, and Democratic Control
Aytaç, U. (2024) · The Journal of Politics · doi:10.1086/729938 · Cited by 28
“One might worry that CBGs are not well positioned to govern Big Tech algorithms even with the help of expert knowledge and adversarial proceedings because the most advanced algorithms can be unexplainable (Castelvecchi 2016).”
Chen, B., Li, Y., Sun, Y., Sun, H., Wang, Y., Lyu, J., Guo, J., Bao, S., Cheng, Y., Niu, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51706 · Cited by 29
“In addition, the inherent “black box” nature of DL models, where decision-making strategies are challenging to understand, has been a common criticism [24,25].”
How do people react to political bias in generative artificial intelligence (AI)?
Messer, U. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100108 · Cited by 43
Oh, S.-W., & Jung, J. (2025) · Media and Communication · doi:10.17645/mac.9495 · Cited by 41
“The situation is further complicated, as even their developers of AI systems may find it difficult to clarify how they function: This is known as the black‐box problem (Castelvecchi, 2016).”
Hamdaoui, K., Benzaamia, A., Sari-Ahmed, B., Guellil, M. E., & Ghrici, M. (2025) · Journal of Engineering and Applied Science · doi:10.1186/s44147-025-00727-4 · Cited by 23
“These models are frequently described as "black box" systems [21], where the internal decision-making processes remain unclear, posing challenges for their application in geotechnical engineering.”
Dexe, J., Franke, U., & Rad, A. (2021) · The Geneva Papers on Risk and Insurance Issues and Practice · doi:10.1057/s41288-021-00207-9 · Cited by 30
“The reason is that even though recent advances in artificial intelligence (AI) have led to impressive results, many algorithms are obscure and ‘black box’-like (Castelvecchi 2016).”
BackgroundIslam, M. M., Poly, T. N., Walther, B. A., Lin, M., & Li, Y. (2021) · Cancers · doi:10.3390/cancers13215253 · Cited by 23
Artificial Intelligence in Interdisciplinary Life Science and Drug Discovery Research
Bajorath, J. (2022) · Future Science OA · doi:10.2144/fsoa-2022-0010 · Cited by 27
“Importantly, similar to other – but not all – ML methods, DNNs have notorious ‘black box’ character [11] , meaning that it is not transparent how these models reach their decisions.”
BackgroundHoang, V.-H., Lee, J. W., Piran, M. J., & Park, C. (2023) · IEEE Access · doi:10.1109/access.2023.3307138 · Cited by 26
Wang, H., Tang, Z., Tan, S. H., Wang, J., Liu, Y., Fang, H., Xia, C., & Wang, Z. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639212 · Cited by 39
“As DL models generally work as a black box [20], to understand Concoction’s workings and the vulnerability’s root cause, we compare the original buggy code with the developer-generated patch after reporting the issue.”
MethodDuan, N., & Stewart, E. (2019) · IET Generation Transmission & Distribution · doi:10.1049/iet-gtd.2018.6195 · Cited by 21
“3 Opening the blackbox for engineering interpretation: Functioning as a blackbox is one widely recognised disadvantage of deep learning algorithms [17].”
BackgroundTriberti, S., Durosini, I., Curigliano, G., & Pravettoni, G. (2020) · Public Health Genomics · doi:10.1159/000506014 · Cited by 27
PAC Model Checking of Black-Box Continuous-Time Dynamical Systems
Bai, X., Zhang, M., Easwaran, A., & Li, Q. (2020) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2020.3012251 · Cited by 21
“systems cannot explain their results even to their makers, let alone to end users [7].”
BackgroundAyensa-Jiménez, J., Doweidar, M. H., Sanz‐Herrera, J. A., & Doblaré, M. (2021) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2021.113816 · Cited by 26
Variational Bayesian calibration of low-cost gas sensor systems in air quality monitoring
Tancev, G., & Toro, F. G. (2021) · Measurement Sensors · doi:10.1016/j.measen.2021.100365 · Cited by 27
Fox, S., & Rey, V. F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030071 · Cited by 22
“[69].”
AI: Can It Make a Difference to the Predictive Value of Ultrasound Breast Biopsy?
Browne, J. L., Pascual, M. Á., Perez, J., Salazar, S., Valero, B., Rodriguez, I., Cassina, D., Alcázar, J. L., Guerriero, S., & Graupera, B. (2023) · Diagnostics · doi:10.3390/diagnostics13040811 · Cited by 20
“KOIOS reaches its rating by processing the data using unknown pathways created by a black box [5], which is so-called because no one knows how it processes the information to deliver its results.”
BackgroundShalaby, A. (2024) · Journal of Economy and Technology · doi:10.1016/j.ject.2024.08.003 · Cited by 39
Soluk, J., Miroshnychenko, I., & Nambisan, S. (2025) · Journal of Product Innovation Management · doi:10.1111/jpim.12789 · Cited by 29
Mao, K., Thu, K. M., Hung, K. F., Yu, O. Y., Hsung, T.-C., & Lam, W. Y. H. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.100883 · Cited by 24
“77,78 Recent efforts have focused on training clinical AI to reason like a team of doctors.”
Applications of Artificial Intelligence in Media and Entertainment
Lachman, R., & Joffe, M. (2021) · Analyzing Future Applications of AI, Sensors, and Robotics in Society · doi:10.4018/978-1-7998-3499-1.ch012 · Cited by 7
Allyn, J., Allou, N., Vidal, C., Renou, A., & Ferdynus, C. (2020) · Medicine · doi:10.1097/md.0000000000023568 · Cited by 23
Syed, S., Ehsan, L., Shrivastava, A., Sengupta, S., Khan, M. N., Kowsari, K., Guleria, S., Sali, R., Kant, K., Kang, S., et al. (2021) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000003057 · Cited by 20
“While these results are noteworthy of their own accord, we further extracted Grad-CAMs to help illuminate the ‘‘black box’’ (22) of these deep learning models, to improve the insight into how the models arrive at their decision-making and provide the much”
BackgroundMethodResultInfluentialGerum, R. C., & Schilling, A. (2021) · Neural Computation · doi:10.1162/neco_a_01424 · Cited by 25
Kim, D., & Lee, J. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2022.109363 · Cited by 28
Lee, J. H., Hwang, E. J., Kim, H., & Park, C. M. (2022) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-1012 · Cited by 25
“…between the model findings and radio-genomics through a gene alteration analysis and gene set enrichment analysis helps readers and researchers understand the biological basis of the model and alleviate its black-box characteristics, which is one of the well-known drawbacks of DL techniques (82).”
Automated Prediction of Kidney Failure in IgA Nephropathy with Deep Learning from Biopsy Images
Testa, F., Fontana, F., Pollastri, F., Chester, J., Leonelli, M., Giaroni, F., Gualtieri, F., Bolelli, F., Mancini, E., Nordio, M., et al. (2022) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.01760222 · Cited by 24
“Post hoc identi fi cation of the features used by deep learning algorithms to carry out their predictions are inherently dif fi cult to identify and, in the deep learning fi eld, the “ black box ” is a well-known problem (26).”
BackgroundLiability Regimes in the Age of AI: a Use-Case Driven Analysis of the Burden of Proof
Fernández Llorca, D., Charisi, V., Hamon, R., Sánchez, I., & Gómez, E. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14565 · Cited by 20
“In the AI domain it is also known as the black-box effect (Castelvecchi, 2016) since the decision-making process with ML appears inscrutable from the outside.”
BackgroundGrote, G. (2023) · Ergonomics · doi:10.1080/00140139.2023.2278408 · Cited by 27
“Self-learning systems such as ChatGPT create the much discussed ‘black-box problem’ (Castelvecchi 2016) referring to the fact that these systems are opaque and unpredictable even for their developers as they autonomously change with any new data available.”
BackgroundNajjar, R. (2024) · Biomedical engineering · doi:10.5772/intechopen.114142 · Cited by 37
Algorithmes de prix, intelligence artificielle et équilibres collusifs
Marty, F. M. (2017) · Revue internationale de droit économique · doi:10.3917/ride.312.0083 · Cited by 26
Luo, X., Chi, W., & Deng, M. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.01145 · Cited by 23
Wang, L., Wang, G., Zhang, M., Fan, D., Liu, X., Guo, Y., Wang, R., Lv, B., Lv, C., Wei, J., et al. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.46 · Cited by 24
Chia, K., Fischer, I., Thomason, P., Graham, H. K., & Sangeux, M. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.529415 · Cited by 26
“This makes a transparent and trustworthy computer system that has been the subject of many research papers (Ferri et al., 2002; Allahyari and Lavesson, 2011; Freitas, 2013; Castelvecchi, 2016; Lipton, 2016; Ribeiro et al., 2016).”
MethodBig Data Analytics and AI in Mental HealthcarePreprint
Rosenfeld, A., Benrimoh, D., Armstrong, C., Mirchi, N., Langlois-Therrien, T., Rollins, C. P. E., Tanguay-Sela, M., Mehltretter, J., Fratila, R., Israel, S., et al. (2019) · arXiv.org · doi:10.48550/arxiv.1903.12071 · Cited by 19
“Furthermore, in most cases, these models turn out to be extremely difficult to interpret beyond a simple understanding of which features were most important in the model [16].”
BackgroundHow recurrent networks implement contextual processing in sentiment analysisPreprint
Maheswaranathan, N., & Sussillo, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08013 · Cited by 29
“Despite this amazing performance, we still only have a rudimentary understanding of exactly how these networks work (Castelvecchi, 2016).”
BackgroundLin, S., Dong, M., Cao, X., Liang, Z., Guo, H., & Zheng, H. (2024) · Engineering Analysis with Boundary Elements · doi:10.1016/j.enganabound.2024.03.019 · Cited by 20
Present and Future Applications of Artificial Intelligence in Kidney Transplantation
Kotsifa, E., & Mavroeidis, V. K. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13195939 · Cited by 25
“Although some scientists compare the black box of AI to the ‘’human black box”, the brain, stating that likewise its functioning remains more or less unknown, many others view AI as a tool to be used with caution [120].”
BackgroundDeep Learning in Digital Breast Tomosynthesis: Current Status, Challenges, and Future Trends
Wang, R., Chen, F., Chen, H., Lin, C. X., Shuai, J., Wu, Y., Ma, L., Hu, X., Wu, M. H., Wang, J., et al. (2025) · MedComm · doi:10.1002/mco2.70247 · Cited by 23
Yazdani, E., Neizehbaz, A., Ziarati, N. K., & Kheradpisheh, S. R. (2025) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2025.102023 · Cited by 23
Explainable artificial intelligence for molecular design in pharmaceutical research
Lamens, A., & Bajorath, J. (2026) · Chemical Science · doi:10.1039/d5sc08461j · Cited by 25
Brain Inspired Sequences Production by Spiking Neural Networks With Reward-Modulated STDP
Fang, H., Zeng, Y., & Zhao, F. (2021) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2021.612041 · Cited by 22
“Second, although the neural network trained by the current deep learning technology can solve some specific problems, its plausibility is very poor (Castelvecchi, 2016), which leads to serious security problems (such as Adversarial Examples Problem) and becomes a black cloud over the head of deep…”
BackgroundDoki, S., Sasahara, S., Hori, D., Oi, Y., Takahashi, T., Shiraki, N., Ikeda, Y., Ikeda, T., Arai, Y., Muroi, K., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-046265 · Cited by 25
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.