Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 401–500 of 1,006
Held, L. A., Wewetzer, L., & Steinhäuser, J. (2022) · Health Informatics Journal · doi:10.1177/14604582221112816 · Cited by 34
“40 Doctors, therefore, could experience dif fi culty in delegating expertise and responsibility for their patients ’ welfare to a machine.”
BackgroundArtificial Intelligence in Medicine: Text Mining of Health Care Workers’ Opinions
Nitiéma, P. (2022) · Journal of Medical Internet Research · doi:10.2196/41138 · Cited by 47
“Using the provided data, the software yields an algorithm that the analyst, manufacturer, and clinician cannot fully understand [29,30].”
Zhang, H., Ni, M., Yang, Y., Xie, F., Wang, W., He, Y., Chen, W., & Chen, Z. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107085 · Cited by 34
Artificial Intelligence and Bias Towards Marginalised Groups: Theoretical Roots and Challenges
Mergen, A., Çetin-Kiliç, N., & Özbilgin, M. F. (2025) · International perspectives on equality, diversity and inclusion · doi:10.1108/s2051-233320250000012004 · Cited by 58
Towards an Explainable Artificial Intelligence approach for smart grid systems
Alfayan, M., & Hagras, H. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00261-5 · Cited by 52
“Moreover, the opacity of the data used to train these models can prevent even the developers from explaining the outcomes generated by the algorithms [9].”
The Applications of Artificial Intelligence and Digital Twin in Power Systems: An In-Depth Review
Rahmani-Sane, G., Azad, S., Ameli, M. T., & Haghani, S. (2025) · IEEE Access · doi:10.1109/access.2025.3580340 · Cited by 46
“Firstly, the internal structure and logic of AI algorithms present challenges regarding explicability and comprehen-sibility, as users can only observe the inputs and outputs without insight into the intervening processes [226].”
Utilizing multimodal artificial intelligence to advance cardiovascular diseases
Yang, X.-Y., Li, Y., Wang, J., Jia, Y., Zhang, Y., & Chen, M. (2025) · Precision Clinical Medicine · doi:10.1093/pcmedi/pbaf016 · Cited by 49
“Comprising millions of parameters and hundreds of layers, the very complexity of AI tools renders them as opaque ‘black-boxes’ [ 88 ], as the computational mechanisms between input and output are too intricate to afford human understanding.”
Lussier, M., Aboujaoudé, A., Couture, M., Moreau, M., Laliberté, C., Giroux, S., Pigot, H., Gaboury, S., Bouchard, K., Belchior, P. D. C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/20215 · Cited by 39
“Transparency is critical for commercial, legal, and clinical applications because professionals must justify their decisions on the basis of tangible observations [38].”
BackgroundEthical issues in computational pathology
Sorell, T., Rajpoot, N., & Verrill, C. L. (2021) · Journal of Medical Ethics · doi:10.1136/medethics-2020-107024 · Cited by 35
“31 Of course, computational pathology does lend itself to the medical ethics norm of informed consent for biopsies that will lead to algorithm-based diagnosis.”
BackgroundNordstrand, J., & Dutta, J. (2021) · npj Clean Water · doi:10.1038/s41545-020-00094-y · Cited by 36
Big Data analytics and artificial intelligence in mental healthcare
Rosenfeld, A., Benrimoh, D. A., Armstrong, C., Mirchi, N., Langlois-Therrien, T., Rollins, C. P. E., Tanguay-Sela, M., Mehltretter, J., Fratila, R., Israel, S. A., et al. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-820203-6.00001-1 · Cited by 77
Deep Weighted MMSE Downlink Beamforming
Pellaco, L., Bengtsson, M., & Jaldén, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414561 · Cited by 36
“These solutions, however, present i) the issue of selecting a suitable architecture, as the search space is still vast despite the architectural constraints, and ii) lack of explainability, as the proposed neural networks are still considered as black boxes [14].”
BackgroundAnanda, A., Ngan, K. H., Karabağ, C., Ter-Sarkisov, A., Alonso, E., & Reyes‐Aldasoro, C. C. (2021) · Sensors · doi:10.3390/s21165381 · Cited by 37
“In addition, one recurrent criticism is their black box nature [52– 55], which implies that it is not always easy or simple to understand how the networks perform in the way they do.”
BackgroundNam, J. G., Park, S., Park, C. M., Jeon, Y. K., Chung, D. H., Goo, J. M., Kim, Y. T., & Kim, H. (2022) · Radiology · doi:10.1148/radiol.213262 · Cited by 32
An eXplainable AI (XAI) model for text-based patent novelty analysis
Jang, H., Kim, S., & Yoon, B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120839 · Cited by 44
Klar, M., Ruediger-Flore, P., Schürmann, M., Gören, G. T., Glatt, M. F., Ravani, B., & Aurich, J. C. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.11.012 · Cited by 42
Lin, Y., Liang, Z., Zhang, H., Zhang, G., Zheng, W., Han, C., Yu, D., Zhang, H., Xie, X., Liu, C., et al. (2024) · Communications Medicine · doi:10.1038/s43856-024-00518-7 · Cited by 47
“Despite the radiologist-level classi fi cation of breast cancer, the remaining pivotal issue that needs to be addressed before its clinical deployment is the black-box nature of deep learning 27,28 .”
Zeng, T., Jiang, H., Liu, Y., & Bai, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126815 · Cited by 36
De‐Giorgio, F., Benedetti, B., Mancino, M., Sala, E., & Pascali, V. L. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112014 · Cited by 35
Conceptualising the emergence of Agentic Urban AI: from automation to agency
Tiwari, A. (2025) · Urban Informatics · doi:10.1007/s44212-025-00079-7 · Cited by 33
“…the recognition of AI’s non-human agency; the iterative and often unpredictable integration of AI systems; the scalar entanglements between local and global infrastructures; and the need to shape progressive alternatives through urban AI development. prioritise urban outcomes (Castelvecchi, 2016).”
Fang, Z., Jin, Y., & Yang, T. (2021) · Journal of Urban Technology · doi:10.1080/10630732.2021.2001713 · Cited by 36
“limited consideration of professional knowledge in guiding urban development (Castelvecchi, 2016; König et al., 2017).”
BackgroundDiagnosis Support Model of Cardiomegaly Based on CNN Using ResNet and Explainable Feature Map
Yoo, H., Han, S., & Chung, K. (2021) · IEEE Access · doi:10.1109/access.2021.3068597 · Cited by 41
“Therefore, most artificial intelligence algorithms being commercialized recently are known as black box models [2], [3].”
A Taxonomy of Food Supply Chain Problems from a Computational Intelligence Perspective
Angarita-Zapata, J. S., Alonso‐Vicario, A., Masegosa, A. D., & Legarda, J. (2021) · Sensors · doi:10.3390/s21206910 · Cited by 42
“…some CI approaches, such as fuzzy logic and ML methods (e.g., rule-based learners, decision trees), can provide different degrees of interpretability and transparency of decisions made in their inner structure; other methods cannot provide this information (e.g., black-box decision systems [158]).”
BackgroundMankodiya, H., Jadav, D., Gupta, R., Tanwar, S., Alharbi, A., Tolba, A. K. R., Neagu, B.-C., & Raboacă, M. S. (2022) · Mathematics · doi:10.3390/math10121990 · Cited by 33
Rodríguez, F., Scheinker, D., & Harrington, R. A. (2018) · Circulation Research · doi:10.1161/circresaha.118.314119 · Cited by 36
“Although AI can yield accurate predictions, the results are often black box by obscuring the importance and selection of predictive inputs.(13) As with all observational data, bias will continue to exist and should be dealt with carefully, but the black box nature of AI algorithms may make it difficult to prevent algo…”
BackgroundZhao, H., Zheng, C., Zhang, H., Rao, M., Li, Y., Fang, D., Huang, J., Zhang, W., & Yuan, G. (2023) · Frontiers in Endocrinology · doi:10.3389/fendo.2023.1224191 · Cited by 39
Gao, Y. J., Da-zhi, L., Sui, Z., & Tian, Y. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3389054 · Cited by 39
“Althougheffortshavebeenmadetoaddresstheissue ofsafetyasnotedin[40],[41],[44],theproblemssuchassam-pleefficiency,complexity,safetyassurance,traininginstability, rewarddesign,interpretability[43]andrewardhacking[42]still exist.”
Hani, I. B., Alawadi, S., & Mrayyan, N. E. (2024) · Journal of Decision System · doi:10.1080/12460125.2024.2349425 · Cited by 48
“This issue inherently makes the decision-making process also not transparent (Castelvecchi, 2016).”
Ebdrup, B. H., Axelsen, M. C., Bak, N., Fagerlund, B., Oranje, B., Raghava, J. M., Nielsen, M. Ø., Rostrup, E., Hansen, L. K., & Glenthøj, B. Y. (2019) · Psychological Medicine · doi:10.1017/s0033291718003781 · Cited by 32
“This overlap in signal provides indirect validation of the applied methods and implies that multivariate algorithms are not a ‘black box’ (Castelvecchi, 2016).”
MethodArtificial intelligence and the hunt for immunological disorders
Rider, N. L., Srinivasan, R., & Khoury, P. (2020) · Current Opinion in Allergy and Clinical Immunology · doi:10.1097/aci.0000000000000691 · Cited by 32
“output are not clearly evident raising suspicion and often leading to mistrust [63].”
BackgroundSlator, P. J., Hutter, J. M., Marinescu, R. V., Palombo, M., Jackson, L. H., Ho, A., Chappell, L. C., Rutherford, M., Hajnal, J. V., & Alexander, D. C. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102045 · Cited by 35
An explainable ensemble feedforward method with Gaussian convolutional filter
Li, J., Shi, H., & Hwang, K. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107103 · Cited by 36
Designing Interactive Transfer Learning Tools for ML Non-Experts
Mishra, S., & Rzeszotarski, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445096 · Cited by 32
“This reliance on intuition may also be attributed to the black-boxed nature of CNN models [11], which has received considerable attention in research community [23].”
BackgroundThe application of artificial intelligence in hepatology: A systematic review
Balsano, C., Alisi, A., Brunetto, M. R., Invernizzi, P., Burra, P., Piscaglia, F., Alvaro, D., Bonino, F., Carbone, M., Faita, F., et al. (2021) · Digestive and Liver Disease · doi:10.1016/j.dld.2021.06.011 · Cited by 34
Yan, C., Chen, Q., Zhou, X., Dai, X., & Yang, Z. (2023) · Journal of Business Ethics · doi:10.1007/s10551-023-05351-x · Cited by 37
“Also, it is often hard to interpret the decisions made by the algorithms because of their complex natures (Castelvecchi, 2016).”
Yang, F., Zuo, R., Xiong, Y., Xu, Y., Nie, J., & Zhang, G. (2024) · Mathematical Geosciences · doi:10.1007/s11004-024-10137-6 · Cited by 35
“Second, the multilayer nonlinear complex structures of DLAs typically prevent researchers from explaining their predictive results, which are extremely dif-ficult to decipher (Castelvecchi 2016; Guidotti et al. 2018; Savage 2022).”
Employees’ foe or friend: artificial intelligence and employee outcomes
Tahir, M. A., Da, G., Javed, M., Akhtar, M. W., & Wang, X. (2024) · Service Industries Journal · doi:10.1080/02642069.2024.2375746 · Cited by 42
“’ Castelvecchi (2016) ‘ De fi ne service AI as the con fi guration of technology to provide value in the internal and external service environments through fl exible adaptation enabled by sensing, learning, decision-making and actions.”
Zhang, K., Wang, D., Lin, F., Xie, J., & Zhou, W. (2026) · iScience · doi:10.1016/j.isci.2026.115026 · Cited by 48
Hettal, L., Stefani, A., Salleron, J., Courrech, F., Behm‐Ansmant, I., Constans, J., Gauchotte, G., & Vogin, G. (2020) · Radiation Research · doi:10.1667/rr15517.1 · Cited by 32
“obscure black boxes without any sense of the phenomenon mechanistics (44).”
BackgroundRoad safety performance function analysis with visual feature importance of deep neural nets
Pan, G., Fu, L., Chen, Q., Yu, M., & Muresan, M. (2020) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003108 · Cited by 31
Deep into Laboratory: An Artificial Intelligence Approach to Recommend Laboratory Tests
Islam, M. M., Poly, T. N., Yang, H., & Li, Y. (2021) · Diagnostics · doi:10.3390/diagnostics11060990 · Cited by 36
Data ownership revisited: clarifying data accountabilities in times of big data and analytics
Fadler, M., & Legner, C. (2021) · Journal of Business Analytics · doi:10.1080/2573234x.2021.1945961 · Cited by 63
“For instance, deep learning techniques operate as “black box” algorithms whose working mechanisms are somehow hard to understand (Castelvecchi, 2016).”
BackgroundGarg, S., Taylor, J., Sherief, M. A. E., Kasson, E. M., Aledavood, T., Riordan, R., Kaiser, N., Cavazos‐Rehg, P. A., & Choudhury, M. D. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100467 · Cited by 44
“5 similarly interpreted due to its complex internal structure (Castelvecchi, 2016).”
BackgroundDong, Y., Jiang, Z., Li, C., Dong, S., Zhang, S., Lv, Y., Sun, F., & Liu, S. (2022) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-980 · Cited by 31
Chinnasamy, P., Sathya, K. B. S., Jebamani, B. J. A., Nithyasri, A., & Fowjiya, S. (2022) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6001-6.ch001 · Cited by 31
Constrained Interval Type-2 Fuzzy Sets
D'Alterio, P., Garibaldi, J. M., John, R. I., & Pourabdollah, A. (2020) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.2970911 · Cited by 31
“However, in the recent years, building explainable intelligent systems has become increasingly important [26], [27].”
BackgroundOn the Decision Boundary of Deep Neural Networks
Li, Y., Ding, L., & Gao, X. (2018) · arXiv · doi:10.48550/arxiv.1808.05385 · Cited by 31
“Despite its great empirical success, deep learning is often criticized for being used as a black box [5], which refers to the well-known gap between its empirical power and the theoretical understanding of it [24].”
Developing an urban streetscape indexing based on visual complexity and self-organizing map
Ma, L., Guo, Z., Lu, M., He, S., & Wang, M. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110549 · Cited by 33
Teng, Y., & Shan, G. (2024) · APL Materials · doi:10.1063/5.0222154 · Cited by 32
A Transparent and Multimodal Malware Detection Method for Android Apps
Zhu, D., Xi, T., Jing, P., Wu, D., Xia, Q., & Zhang, Y. (2019) · ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM) · doi:10.1145/3345768.3355915 · Cited by 30
“Therefore, researchers can only obtain a certain decision from the neural nets but don’t know what caused it [6].”
BackgroundChen, J.-R., Chao, Y., Tsai, Y.-W., Chan, H.-J., Wan, Y., Tai, D., & Tsui, P. (2020) · Entropy · doi:10.3390/e22091006 · Cited by 33
Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings
Turton, J., Smith, R. E., & Vinson, D. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.26 · Cited by 46
Artificial Intelligence shaping the future of neurology practice
Vinny, P. W., Vishnu, V. Y., & Srivastava, M. V. P. (2021) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2021.06.003 · Cited by 33
Angarita-Zapata, J. S., Maestre-Gongora, G. P., & Calderín, J. F. (2021) · Sensors · doi:10.3390/s21248401 · Cited by 42
“While ML methods (e.g., rule-based learners, decision trees) can provide different degrees of interpretability of decisions made in the inner structure of the methods, AutoML and AutoDL approaches cannot usually offer this information [78].”
MethodWang, P., Xiao, J., Kawaguchi, K., & Wang, L. (2022) · Sustainability · doi:10.3390/su14063275 · Cited by 33
Affective Design Analysis of Explainable Artificial Intelligence (XAI): A User-Centric Perspective
Bernardo, E. L., & Seva, R. R. (2023) · Informatics · doi:10.3390/informatics10010032 · Cited by 48
YANIT, M., Yanit, M., & Wan, F. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107870 · Cited by 32
How will ai transform urban observing, sensing, imaging, and mapping?
Weng, Q., Li, Z., Cao, Y., Lu, X., Gamba, P. E., Zhu, X. X., Xu, Y., Zhang, F., Qin, R., Yang, M. Y., et al. (2024) · npj Urban Sustainability · doi:10.1038/s42949-024-00188-3 · Cited by 41
“AI models often operate as a “ black box ” 21 , neglecting the underlying physical mechanisms.”
Machine learning prediction of materials properties from chemical composition: Status and prospects
Alghadeer, M., Aisyah, N. D., Hezam, M., Alqahtani, S. M., Baloch, A. A. B., & Alharbi, F. H. (2024) · Chemical Physics Reviews · doi:10.1063/5.0235541 · Cited by 37
Lelescu, A., Sava, S., Grosseck, G., & Maliţa, L. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06253-1 · Cited by 54
“This caution exists due to various concerns regarding the complexity of AI systems (Castelvecchi, 2016), trust issues (Gil-lespie et al., 2023), and insuf fi cient institutional support (Duah & McGivern, 2024; Gimpel et al., 2023).”
A review of explainable AI methods and their application in manufacturing systems
Tzionis, G., Mouratidis, P., Kougka, G., Gialampoukidis, I., Vrochidis, S., Kompatsiaris, I. Y., & Vlachopoulou, M. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07908-z · Cited by 50
Feldmann, C., & Bajorath, J. (2022) · iScience · doi:10.1016/j.isci.2022.105023 · Cited by 32
“Most non-linear MLmethods have a black box character (Castelvecchi, 2016) meaning that their predictions cannot be intuitively accessed and understood by humans.”
Deep-spindle: An automated sleep spindle detection system for analysis of infant sleep spindles
Wei, L., Ventura, S., Ryan, M. A., Mathieson, S., Boylan, G. B., Lowery, M. M., & Mooney, C. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106096 · Cited by 31
Yao, L., Lu, Z., Yang, G., Zhou, W., Xu, Y., Guo, M., Huang, X., He, C., Zhou, R., Deng, Y., et al. (2022) · Digestive Endoscopy · doi:10.1111/den.14493 · Cited by 31
“16 The opaqueness of the diagnosis process may weaken an endoscopist ’ s con fi dence in the results of AI studies to predict the extent of CRC deep invasions.”
BackgroundXia, Q., Lee, F., & Chen, Q. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119008 · Cited by 29
People Analytics im Personalmanagement: Auf dem Weg zur automatisierten Entscheidungskultur?
Kels, P., & Vormbusch, U. (2020) · Industrielle Beziehungen Zeitschrift für Arbeit Organisation und Management · doi:10.3224/indbez.v27i1.04 · Cited by 32
“Auch für vorgebildete Laien wie betriebliche Experten, Betriebsräte oder Personalmanager sind sie brain“ (Castelvecchi, 2016, S. 21 vollkommen intransparente ‚black boxes‘, „as opaque as the ).”
Artificial Intelligence in Hepatology- Ready for the Primetime
Kalapala, R., Rughwani, H., & Reddy, D. N. (2022) · Journal of Clinical and Experimental Hepatology · doi:10.1016/j.jceh.2022.06.009 · Cited by 30
TAG-DTA: Binding-region-guided strategy to predict drug-target affinity using transformers
Monteiro, N. R. C., Oliveira, J. L., & Arrais, J. P. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122334 · Cited by 28
DKNN: deep kriging neural network for interpretable geospatial interpolation
Chen, K., Liu, E., Deng, M., Tan, X., Wang, J., Shi, Y., & Wang, Z. (2024) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2024.2347316 · Cited by 39
“However, while benefiting from the powerful capabilities of deep learning, the above models also inherit the disadvantages of their ‘black-box’ process, that is, poor interpretability and high computational expense (Castelvecchi 2016).”
An explainable dual-mode convolutional neural network for multivariate time series classification
Cai, W., Zhu, X., Bai, K., Ye, A., & Zhang, R. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112015 · Cited by 35
Harnessing the Missing Spectral Correlation for Metasurface Inverse Design
Zhang, J., Qian, C., You, G., Wang, T., Saifullah, Y., Abdi–Ghaleh, R., & Chen, H. (2024) · Advanced Science · doi:10.1002/advs.202308807 · Cited by 32
The Rise of Agentic AI: Synthesis of Current Knowledge and Future Research Agenda
Islam, M. A., Somu, S., & Aldaihani, F. M. F. (2025) · Global Business and Organizational Excellence · doi:10.1002/joe.70019 · Cited by 58
Fenech, M. E., & Buston, O. (2020) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2020.00054 · Cited by 37
“The “black box” problem is one of the major foci of AI ethics (37).”
BackgroundMaloca, P. M., Müller, P. L., Lee, A., Tufail, A., Balaskas, K., Niklaus, S., Kaiser, P., Suter, S. K., Zarranz‐Ventura, J., Egan, C. A., et al. (2021) · Communications Biology · doi:10.1038/s42003-021-01697-y · Cited by 29
Zhang, Q., Du, Q., & Liu, G. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac37cc · Cited by 33
“Fourth, the AI model has black box characteristics, and its inter-pretability is still facing challenges [19, 20].”
Sleep as a random walk: a super-statistical analysis of EEG data across sleep stages
Metzner, C., Schilling, A., Traxdorf, M., Schulze, H., & Krauß, P. (2021) · Communications Biology · doi:10.1038/s42003-021-02912-6 · Cited by 33
Sankari, V. M. R., Umapathy, S., Chandrasekaran, A., & Baskaran, P. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104883 · Cited by 30
Engineering Challenges of Stationary Wireless Smart Ocean Observation Systems
Nguyen, N.-T., Heldal, R., Lima, K., Oyetoyan, T. D., Pelliccione, P., Kristensen, L. M., Høydal, K. W., Reiersgaard, P. A., & Kvinnsland, Y. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3283252 · Cited by 32
Elite male table tennis matches diagnosis using SHAP and a hybrid LSTM–BPNN algorithm
Song, H., Li, Y., Zou, X., Hu, P., & Liu, T. (2023) · Scientific Reports · doi:10.1038/s41598-023-37746-1 · Cited by 31
Sebastian, G., & Sebastian, S. S. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4461801 · Cited by 41
Trust and Transparency in Machine Learning-Based Clinical Decision Support
Gretton, C. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_14 · Cited by 27
21st Century teaming and beyond: Advances in human-autonomy teamwork
O’Neill, T., Flathmann, C., McNeese, N. J., & Salas, E. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107865 · Cited by 35
Agrawal, G. (2024) · Advances in healthcare information systems and administration book series · doi:10.4018/979-8-3693-7452-8.ch009 · Cited by 47
Karger, E., Rothweiler, A., Brée, T., & Ahlemann, F. (2025) · Urban Science · doi:10.3390/urbansci9040132 · Cited by 35
Artificial Intelligence in Cardiac Electrophysiology: A Comprehensive Review
Cipollone, P., Pierucci, N., Matteucci, A., Palombi, M., Laviola, D., Bruti, R. M., Vinciullo, S., Bernardi, M., Spadafora, L., Cersosimo, A., et al. (2025) · Journal of Personalized Medicine · doi:10.3390/jpm15110532 · Cited by 35
“An increasing number of experts in the field are questioning how to make neural networks more transparent and black boxes more accessible [43].”
The Responsibility Quantification Model of Human Interaction With Automation
Douer, N., & Meyer, J. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2965466 · Cited by 31
“possible behaviors and outcomes, since their internal structure can be opaque (a “black box”) and sometimes can yield odd and counterintuitive results [25], [26].”
Oliver, D., Wong, C. M. J., Bøg, M., Jönsson, L., Kinon, B. J., Wehnert, A., Jørgensen, K. T., Irving, J., Stahl, D., McGuire, P., et al. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-01032-9 · Cited by 26
Employing machine learning techniques to assess requirement change volatility
Hein, P. H., Kames, E., Chen, C., & Morkos, B. W. (2021) · Research in Engineering Design · doi:10.1007/s00163-020-00353-6 · Cited by 35
“It is important to note that many of the solutions observed here are the classical AI black box problem (Castelvecchi 2016; Bathaee 2018), where a solution is realized through the analysis, but the authors have yet to recognize how each metric influences the output.”
BackgroundPan, Q., Hu, W., & Chen, N. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/397 · Cited by 36
“Though effective in improving forecasting accuracy, deep models are hard to interpret the outputs [Castelvecchi, 2016], which may hinder their applications to high stakes applications (e.g., healthcare) where reliable interpretation is crucial.”
BackgroundDeep Machine Learning for Computer-Aided Drug Design
Bajorath, J. (2022) · Frontiers in Drug Discovery · doi:10.3389/fddsv.2022.829043 · Cited by 32
“Shallow NNs were popular in chemoinformatics during the early days of ML, but have largely been replaced over the years with other ML methods, mostly due to the tendency of NNs to over fi t models for small data sets and their notorious black box character (Castelvecchi, 2016).”
BackgroundOpportunities for pharmacoproteomics in biomarker discovery
Poulos, R. C., Cai, Z., Robinson, P. J., Reddel, R. R., & Zhong, Q. (2022) · PROTEOMICS · doi:10.1002/pmic.202200031 · Cited by 30
“a black box due to their relatively limited model explainability [50].”
BackgroundGu, J., Pan, J., Hu, J., Dai, L., Zhang, K., Wang, B., He, M., Zhao, Q., & Jiang, T. (2023) · Cancer · doi:10.1002/cncr.34772 · Cited by 29
“However, the black‐box nature of deep learningmakes it difficult for it to be widely used in clinical practice.(41,42) To overcome this barrier, we introduced interpretable features, that is, heatmaps.”
BackgroundINTELIGÊNCIA ARTIFICIAL E EDUCAÇÃO: COMO A IA ESTÁ MUDANDO A MANEIRA COMO APRENDEMOS E ENSINAMOS
Picão, F. F., Gomes, L. F., Alves, L., Barpi, O., & Luccheti, T. A. (2023) · Revista Amor Mundi · doi:10.46550/amormundi.v4i5.254 · Cited by 27
Producing radiologist-quality reports for interpretable artificial intelligencePreprint
Gale, W., Oakden‐Rayner, L., Carneiro, G., Bradley, A. P., & Palmer, L. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00340 · Cited by 46
“In response to these concerns, researchers have sought to “open the black box of AI” [5] by attempting to explain the decisions of systems in a variety of ways, which typically have fallen into the following categories: 1) identifying regions of the image that contribute to the decision (saliency mapping/heatmaps) [6]…”
BackgroundExploring explainable AI features in the vocal biomarkers of lung disease
Chen, Z., Liang, N., Li, H., Zhang, H., Li, H., Yan, L., Hu, Z., Chen, Y., Zhang, Y., Wang, Y., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108844 · Cited by 26
Ghandour, A., Woodford, B. J., & Abusaimeh, H. (2024) · IEEE Access · doi:10.1109/access.2024.3394243 · Cited by 35
“This lack of transparency can lead to distrust and accountability issues, particularly when the AI’s decisions have a significant impact [33].”
BackgroundResultQuality work in the future: New directions via a co-evolving sociotechnical systems perspective
Parker, S. K., Ballard, T., Billinghurst, M., Collins, C. G., Dollard, M. F., Griffin, M. A., Johal, W., Jorritsma, K., Kowalkiewicz, M., Kyndt, E., et al. (2025) · Australian Journal of Management · doi:10.1177/03128962251331813 · Cited by 52
“Crucially, the opaqueness of these systems reduces the possibility for humans to actively control the technology (Castelvecchi, 2016), with autonomy by workers being a crucial principle of sociotechnical systems theory and integral to the idea of ‘human in the loop’.”
Recent advances and future research directions in deep learning as applied to geochemical mapping
Xu, Y., Zuo, R., Chen, Z., Shi, Z., & Kreuzer, O. P. (2025) · Earth-Science Reviews · doi:10.1016/j.earscirev.2025.105209 · Cited by 32
Objective Estimation of Sensory Thresholds Based on Neurophysiological Parameters
Schilling, A., Gerum, R. C., Krauß, P., Metzner, C., Tziridis, K., & Schulze, H. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00481 · Cited by 35
“However, machine learning approaches generate black-box systems with complex internal decision criteria that are not comprehensive to the users (Castelvecchi, 2016).”
BackgroundMeasure Utility, Gain Trust: Practical Advice for XAI Researchers
Davis, B., Glenski, M., Sealy, W. I. N., & Arendt, D. (2020) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex51495.2020.00005 · Cited by 43
“Here, explanations are used to provide insight into the mechanisms of complex “black box” models, e.g., neural networks or other deep learning models [9], to identify flaws or biases in the algorithm or the training data that can be addressed in development.”
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.