Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 601–700 of 1,006
aiSTROM–A Roadmap for Developing a Successful AI Strategy
Herremans, D. (2021) · IEEE Access · doi:10.1109/access.2021.3127548 · Cited by 41
“This concept is also referred to as XAI [78], [79].”
BackgroundArtificial Intelligence and Deep Learning in Ophthalmology
Wang, Z., Keane, P. A., Chiang, M. F., Cheung, C. Y.-L., Wong, T. Y., & Ting, D. S. W. (2021) · Springer eBooks · doi:10.1007/978-3-030-58080-3_200-1 · Cited by 30
Two Sides of the Same Coin: Exploiting the Impact of Identifiers in Neural Code Comprehension
Gao, S., Gao, C., Wang, C., Sun, J., Lo, D., & Yu, Y. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00164 · Cited by 39
Feature importance measure of a multilayer perceptron based on the presingle-connection layer
Zhang, W., Shen, X., Zhang, H., Yin, Z., Sun, J., Zhang, X., & Zou, L. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01959-7 · Cited by 23
“Consequently, it is challenging to justify the reliability of the model in solving real-world problems [43], especially in high-risk areas [44, 45].”
Albarracin, M., Hipólito, I., Tremblay, S. E., Fox, J. G., René, G., Friston, K. J., & Ramstead, M. J. D. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47958-8_9 · Cited by 26
Wang, T., Hong, J., Chiu, H., Chao, H., Chen, Y., & Wu, Y. (2024) · European Radiology · doi:10.1007/s00330-024-10804-6 · Cited by 27
“Radiologists concerned about the “ black box ” nature of AI models question their transparency and analytical ability in clinical decision-making [7].”
Explainable AI-Empowered Neuromorphic Computing Framework for Consumer Healthcare
Sai, S., Sharma, S., & Chamola, V. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3438160 · Cited by 23
“Many models, although offering very high accuracy and performance and thus making very accurate fitting to the pattern of the data, often become too complex and unexplainable, referred to as “Black Boxes” [8].”
Data & Policy: A new venue to study and explore policy–data interaction
Verhulst, S., Engin, Z., & Crowcroft, J. (2019) · Data & Policy · doi:10.1017/dap.2019.2 · Cited by 22
“A number of associated challenges are being discussed, such as: (i) traceability and reproducibility of research outcomes (due to “black box processing”); (ii) the use of correlation instead of causation as the basis of analysis, biases and uncertainties present in large historical datasets that cause replication and,…”
BackgroundLess may be more: an informed reflection on molecular descriptors for drug design and discovery
Barnard, T., Hagan, H., Tseng, S., & Sosso, G. C. (2019) · Molecular Systems Design & Engineering · doi:10.1039/c9me00109c · Cited by 23
Superimposition: Augmenting Machine Learning Outputs with Conceptual Models for Explainable AI
Lukyanenko, R., Castellanos, A., Storey, V. C., Castillo, A., Tremblay, M. C., & Parsons, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65847-2_3 · Cited by 23
Kang, B., Yuan, X., Wang, H., Qin, S., Song, X., Yu, X., Zhang, S., Sun, C., Zhou, Q., Wei, Y., et al. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.750875 · Cited by 23
Jaigirdar, F. T., Rudolph, C., Oliver, G., Watts, D., & Bain, C. (2020) · 2020 IEEE 6th International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic50333.2020.00030 · Cited by 18
“There are repeated examples [9] of the problematic nature of this type of black-box decision-making that points to inaccurate and unsafe decision-making as well as instances of bias, discrimination, lack of due process, and breaches of human rights.”
BackgroundCheung, J. C. T., & Ho, S. S. (2024) · Public Understanding of Science · doi:10.1177/09636625241291192 · Cited by 23
“Indeed, the domain of engineering has witnessed a profound XAI investigation trend, largely because the “black box” phenomenon poses more challenges to AI engineers, especially with the advancement in machine learning and deep learning capabilities (Castelvecchi, 2016).”
Yang, Y., Li, H., Lei, Z., Yang, H., & Wang, J. (2024) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2024.101832 · Cited by 21
ShamAIn: Designing Superior Conversational AI Inspired by Shamanism
Cho, H., Seo, J. A., Lee, J., Kim, C.-M., & Nam, T.-J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714297 · Cited by 24
“This phenomenon is often attributed to the uncertainty and unpredictability of AI systems [14, 16], which pose challenges for HCI researchers [57, 105].”
Srivastava, M., Nandan, S., Zaidi, A., Samani, A. S., Shukla, V., Aslam, H., Srivastava, A., Maurya, P., Khan, M. A., Khan, M. F., et al. (2025) · ChemistrySelect · doi:10.1002/slct.202404446 · Cited by 28
Sun, L., Tang, Y., & Ma, X. (2025) · European Journal of Marketing · doi:10.1108/ejm-02-2023-0082 · Cited by 28
AI and Its Implications for Organisations
Sharma, M., & Biros, D. P. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83909-812-320211001 · Cited by 23
Lai, H., Fu, S., Zhang, J., Cao, J., Feng, Q., Lu, L., & Huang, M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167788 · Cited by 19
“whole image in the CNN for classification [12].”
MethodMerkaj, S., Bahar, R. C., Zeevi, T., Lin, M., Ikuta, I., Bousabarah, K., Petersen, G. C., Staib, L. H., Payabvash, S., Mongan, J., et al. (2022) · Cancers · doi:10.3390/cancers14112623 · Cited by 20
Explainable Artificial Intelligence in Genomic Sequence for Healthcare Systems Prediction
Awotunde, J. B., Adeniyi, A. E., Ajamu, G. J., Balogun, G. B., & Taofeek-Ibrahim, F. A. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-97929-4_19 · Cited by 24
Lupo, G. (2022) · Oñati Socio-legal Series · doi:10.35295/osls.iisl/0000-0000-0000-1273 · Cited by 28
Artificial intelligence in cardiac magnetic resonance fingerprinting
Velasco, C., Fletcher, T. J., Botnar, R. M., & Prieto, C. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.1009131 · Cited by 21
“However, this so-called black-box problem (121) still needs to be addressed so that the community can better comprehend what factors contribute to the decision-making of a ML model in order to better interpret its outputs.”
BackgroundLu, W., Qiu, T., Shi, W., & Sun, X. (2022) · Complexity · doi:10.1155/2022/1511479 · Cited by 22
“However, the neural network models have transparency problem and it is difcult to understand the economic implications and impact mechanisms [10].”
BackgroundThe Threat of Adversarial Attack on a COVID-19 CT Image-Based Deep Learning System
Li, Y., & Liu, S. (2023) · Bioengineering · doi:10.3390/bioengineering10020194 · Cited by 23
“More unexpectedly, the same image perturbation was shown to fool many network classifiers, and this interesting phenomenon has encouraged many researchers to pay more attention to issues such as the security of deep learning [37,38].”
BackgroundArtificial intelligence to enhance corporate governance: A conceptual framework
Correia, A. C., & Água, P. B. (2023) · Corporate Board role duties and composition · doi:10.22495/cbv19i1art3 · Cited by 24
“Moreover, the decisionmaking process of AI systems can be opaque, making it difficult for humans to understand and explain the decisions, which can undermine accountability and trust in AI systems (Castelvecchi, 2016).”
BackgroundMethodInfluentialMulti-Network-Based Ensemble Deep Learning Model to Forecast Ross River Virus Outbreak in Australia
Sakib, M., & Siddiqui, T. (2023) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001423520158 · Cited by 18
Artificial Intelligence (AI) and Cheating: The Concept of Generative Artificial Intelligence (GenAI)
Akkaş, Ö. M., Tosun, C., & Gökçearslan, Ş. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1351-0.ch009 · Cited by 22
Liao, N.-Q., Deng, Z., Wei, W., Lü, J., Li, M., Ma, L., Chen, Q., & Zhong, J. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.001 · Cited by 21
“[40] Research indicates that models offering interpretability not only boost clinicians ’ confidence in diagnosis but also provide valuable educational feedback, benefiting those less specialized, such as Studies have revealed that necrosis is a constituent of complex intratumoral components that…”
BackgroundLi, E., Zhu, E., Xiao, T., Bao, W., Wei, Z., Wang, Q., Qin, F., & Yin, W.-Y. (2024) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2024.3384064 · Cited by 20
“To provide the confidence of the predicted results, BNN is introduced to qualify the uncertainty of the predicted results in inverse problems [45], [46], [47].”
MethodExplainable AI for Earth observation: current methods, open challenges, and opportunities
Taşkın, G., Aptoula, E., & Ertürk, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-319077-3.00012-2 · Cited by 24
Interpretable machine learning with reject option
Brinkrolf, J., & Hammer, B. (2018) · at - Automatisierungstechnik · doi:10.1515/auto-2017-0123 · Cited by 21
Constrained Interval Type-2 Fuzzy Classification Systems for Explainable AI (XAI)
D'Alterio, P., Garibaldi, J. M., & John, R. I. (2020) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz48607.2020.9177671 · Cited by 18
“56, obtained because NPI IS High [1, 1] AND ER IS Weak [0.”
BackgroundMethodInfluentialEngel, C., Ebel, P., & van Giffen, B. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_12 · Cited by 45
“(3) Black box character refers to AI systems, especially in the field of deep learning, facing challenges in delivering explanations to humans on what happens between data input and AI output [16].”
BackgroundChen, Q., Pan, G., Chen, W., & Wu, P. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3084846 · Cited by 23
“To address these limitations, researchers have begun exploring a new area, XAI, to strengthen DL models [15]–[19].”
BackgroundTabas, S. S., & Samadi, V. S. (2024) · Journal of Water Resources Planning and Management · doi:10.1061/jwrmd5.wreng-6089 · Cited by 17
Biswas, S., Grundlingh, N., Boardman, J., White, J., & Le, L. (2025) · Electronics · doi:10.3390/electronics14030571 · Cited by 20
Pue, J. D., Botula, Y.-D., Nguyen, P. M., Meirvenne, M. V., & Cornelis, W. (2020) · Journal of Hydrology · doi:10.1016/j.jhydrol.2020.125770 · Cited by 17
A Checklist for Explainable AI in the Insurance Domain
Koster, O., Kosman, R., & Visser, J. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_32 · Cited by 18
Gili, T., Carlo, G. D., Capuani, S., Auconi, P., Caldarelli, G., & Polimeni, A. (2021) · Orthodontics and Craniofacial Research · doi:10.1111/ocr.12520 · Cited by 19
Ultrafast simulation of large-scale neocortical microcircuitry with biophysically realistic neurons
Oláh, V. J., Pedersen, N. P., & Rowan, M. J. (2022) · eLife · doi:10.7554/elife.79535 · Cited by 15
“Due to the opaque nature of neural net operations (Castelvecchi, 2016), it is reasonable to assume that instant modification of the trained architecture to account for specific biophysical alterations may not be feasible, highlighting a potentially significant shortcoming of our approach.”
BackgroundMethodOptimizing Resource Allocation Based on Predictive Process Monitoring
Park, G., & Song, M. (2023) · IEEE Access · doi:10.1109/access.2023.3267538 · Cited by 17
“The explainability of predictive models has been more relevant due to the nature of neural network-based predictive models that the learned structure is difficult to intuitively represent, known as the black-box problem [17].”
BackgroundCombining machine learning and molecular simulations to predict the stability of amorphous drugs
Barnard, T., & Sosso, G. C. (2023) · The Journal of Chemical Physics · doi:10.1063/5.0156222 · Cited by 18
Maidana, R. G., Parhizkar, T., Martín, G. S., & Utne, I. B. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109725 · Cited by 21
Ghost diffractive deep neural networks: Optical classifications using light’s second-order coherence
Ye, Z., Zhou, C., Ding, C., Zhao, J., Jiao, S., Wang, H.-B., & Xiong, J. (2023) · Physical Review Applied · doi:10.1103/physrevapplied.20.054012 · Cited by 19
Detecting DDoS based on attention mechanism for Software-Defined Networks
Yoon, N., & Kim, H. (2024) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2024.103928 · Cited by 18
Roth, J. P., & Bajorath, J. (2024) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2024.102110 · Cited by 26
Venous thromboembolism in the era of machine learning and artificial intelligence in medicine
Gil, M. R., Pantanowitz, J., & Rashidi, H. H. (2024) · Thrombosis Research · doi:10.1016/j.thromres.2024.109121 · Cited by 19
Regulating Explainable Artificial Intelligence (XAI) May Harm Consumers
Mohammadi, B., Malik, N., Derdenger, T. P., & Srinivasan, K. (2024) · Marketing Science · doi:10.1287/mksc.2022.0396 · Cited by 35
An interdisciplinary perspective on AI-supported decision making in medicine
Ammeling, J., Aubreville, M., Fritz, A., Kießig, A., Krügel, S., & Uhl, M. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102791 · Cited by 22
Algorithmic transparency of conversational agents
Hepenstal, S., Kodagoda, N., Zhang, L., Paudyal, P., & Wong, B. L. W. (2019) · Middlesex University Research Repository (Middlesex University Of London) · Cited by 24
Gargantilla-Becerra, Á., Gutiérrez, M., & Lahoz-Beltrá, R. (2022) · Biosystems · doi:10.1016/j.biosystems.2022.104608 · Cited by 26
Punishing Artificial Intelligence: Legal Fiction or Science Fiction
Abbott, R. B., & Sarch, A. (2024) · Data science, machine intelligence, and law · doi:10.1007/978-3-031-47946-5_6 · Cited by 22
Scarcello, F. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20326-9 · Cited by 20
Hristova, T., Magee, L., & Soldatić, K. (2024) · AI & Society · doi:10.1007/s00146-024-02039-2 · Cited by 35
“First, the system is an example of a black box with mechanisms of operation that remain hidden behind technical complexity and proprietary enclosures of information (Castelvecchi 2016).”
An artificial neural network based method to uncover the value-of-travel-time distribution
van Cranenburgh, S., & Kouwenhoven, M. (2020) · Transportation · doi:10.1007/s11116-020-10139-3 · Cited by 17
“…such as the relative importance of attributes, Willingness-to-Pay, or VTT. 1 3 This holds especially true for ANNs, which are widely considered black boxes (Castelvecchi 2016), amongst other things, because it is impossible to interpret or diagnose ANNs by looking at the weights of the network.”
BackgroundStructure–activity relationships and drug design
Gianti, E., & Zauhar, R. J. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-820007-0.00007-6 · Cited by 19
Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture
Mitsopoulos, K., Somers, S., Schooler, J. N., Lebiere, C. J., Pirolli, P. L. T., & Thomson, R. (2021) · Topics in Cognitive Science · doi:10.1111/tops.12573 · Cited by 17
Korjian, S., & Gibson, C. M. (2022) · Contemporary Clinical Trials · doi:10.1016/j.cct.2022.106767 · Cited by 19
Adapting a Trusted AI Framework to Space Mission Autonomy
Slingerland, P. C., Perry, L., Kaufman, J. C., Bycroft, B. P., Linstead, E. J., Mandrake, L., Doran, G. B., Goel, A., Feather, M. S., Fesq, L. M., et al. (2022) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero53065.2022.9843376 · Cited by 19
Ohno, Y., Mazaki, J., Udo, R., Tago, T., Kasahara, K., Enomoto, M., Ishizaki, T., & Nagakawa, Y. (2022) · Cancer Diagnosis & Prognosis · doi:10.21873/cdp.10161 · Cited by 16
Joy, S. I., Kumar, K. S., Palanivelan, M., & Lakshmi, D. (2023) · Canadian Journal of Electrical and Computer Engineering · doi:10.1109/icjece.2022.3228588 · Cited by 22
“The transparency establishment in human understanding about computer intelligence continues to be a big challenge [50].”
BackgroundExtracting continuous sleep depth from EEG data without machine learning
Metzner, C., Schilling, A., Traxdorf, M., Schulze, H., Tziridis, K., & Krauß, P. (2023) · Neurobiology of Sleep and Circadian Rhythms · doi:10.1016/j.nbscr.2023.100097 · Cited by 20
“However, the typical automatic classifiers used in the field of machine learning are black boxes with huge numbers of parameters, which makes their classification decisions hard to interpret and difficult to reproduce [9, 10].”
BackgroundMitigating Algorithm Aversion in Recruiting: A Study on Explainable AI for Conversational Agents
Fleiß, J., Bäck, E., & Thalmann, S. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3645057.3645062 · Cited by 25
“It is well known that opaque information systems (IS), such as black box AI-based systems, have a negative effect on user acceptance (Castelvecchi, 2016).”
BackgroundMorton, F., Benavides, T. B. T., & González-Treviño, E. (2024) · Management and industrial engineering · doi:10.1007/978-3-031-52990-0_2 · Cited by 25
An extensive bibliometric analysis of artificial intelligence techniques from 2013 to 2023
Bajpai, A., Yadav, S., & Nagwani, N. K. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07021-3 · Cited by 24
“During the 1990s, there was a period known as the “AI winter,” characterized by a decrease in financial backing and excitement for AI research, resulting in a downturn in several AI research domains [38].”
Pizzolla, I., Aro, R., Duez, P., Lièvre, B. D., & Briganti, G. (2023) · The Journal of Interactive Learning Research · doi:10.70725/265312mqoebd · Cited by 26
“Besides, the “black box” nature of certain AI algorithms may challenge traditional notions of medical accountability, as it may become difficult to determine responsibility in cases of misdiagnosis or treatment errors (Castelvecchi, 2016; Nicholson Price II, 2014).”
BackgroundIntelligence clinique et intelligence artificielle: Une question de nuance
Matuchansky, C. (2019) · médecine/sciences · doi:10.1051/medsci/2019158 · Cited by 26
Regulatory issues for artificial intelligence in radiology
Pesapane, F., Suter, M. B., Codari, M., Patella, F., Volonté, C., & Sardanelli, F. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-819178-1.00052-6 · Cited by 20
The Evaluation of the Black Box Problem for AI-Based Recommendations: An Interview-Based Study
Ochmann, J., Zilker, S., & Laumer, S. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_16 · Cited by 25
Burlacu, A., Crișan-Dabija, R. A., Popa, I. V., Artene, B., Birzu, V., Pricop, M., Pleșoianu, C., & Generali, D. (2020) · medRxiv · doi:10.1101/2020.04.28.20082776 · Cited by 16
“The “black-box” issue, firstly spotlighted by Dean Pomerleau in 1991, has become exponentially harder and more urgent (3).”
BackgroundMethodLamens, A., & Bajorath, J. (2025) · Molecular Informatics · doi:10.1002/minf.202500067 · Cited by 16
Wu, H., Zhang, Q., Zhu, J., Wu, L., Xiao, Y., & Yang, X. (2025) · Bioactive Materials · doi:10.1016/j.bioactmat.2025.09.002 · Cited by 35
“First, the “ black box ” nature of many complex AI models presents a significant hurdle for clinical translation, where mechanistic understanding and model transparency are critical for regulatory approval and risk assessment [329 – 331].”
Song, Y., & Zhang, A. (2025) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2025.09.003 · Cited by 34
Booth, B. G., Hoefnagels, E. M., Huysmans, T., Sijbers, J., & Keijsers, N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229685 · Cited by 17
“However, these machine learning techniques often appear to users as a black box: it is unclear how the algorithm is making its choice [30].”
BackgroundYi, J., Shin, Y., Hahn, S., & Lee, Y. H. (2022) · Scientific Reports · doi:10.1038/s41598-022-07683-6 · Cited by 20
Hamilton, A., Characklis, G. W., & Reed, P. M. (2021) · Water Resources Research · doi:10.1029/2021wr029747 · Cited by 15
“However, decision-makers may be unwilling to adopt a complex, non-linear control policy if its operating rules remain opaque; it may be necessary to “open the black box” for users if they are to apply such tools in practice (Castelvecchi, 2016; Quinn et al., 2019).”
BackgroundHayakawa, J., Seki, T., Kawazoe, Y., & Ohe, K. (2022) · PLoS ONE · doi:10.1371/journal.pone.0269570 · Cited by 18
“The problem is often referred to as the black box problem in deep learning [1].”
BackgroundHigh-resolution 3D mapping of cold-water coral reefs using machine learning
de Oliveira, L. M. C., Lim, A., Conti, L. A., & Wheeler, A. J. (2022) · Frontiers in Environmental Science · doi:10.3389/fenvs.2022.1044706 · Cited by 24
“Additionally, researchers are continuously working towards minimising the effects of the “ black box issue ” to properly understand the machine decision process in the classi fi cation and how parameters and data affect the output (Gilardi 1995; Castelvecchi 2016).”
BackgroundJacobs, E., Wainman, B. C., & Bowness, J. S. (2023) · Anatomical Sciences Education · doi:10.1002/ase.2266 · Cited by 18
“41 By presenting the convolutional neural network with large volumes of input data, the network will learn to produce a fine-grained analysis of the original image.”
BackgroundAn Alternative to Cognitivism: Computational Phenomenology for Deep Learning
Beckmann, P., Köstner, G., & Hipólito, I. (2023) · Minds and Machines · doi:10.1007/s11023-023-09638-w · Cited by 32
“For this reason they are sometimes referred to as “black boxes” (Castelvecchi, 2016).”
BackgroundStudy the correlation between the readme file of GitHub projects and their popularity
Wang, T., Wang, S., & Chen, T.-H. P. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111806 · Cited by 35
Huang, H., Han, L., Guo, J., Zhang, Y., Lin, S., Chen, S., Lin, X., Cheng, C., Guo, Z., & Qiu, Y. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.10.055 · Cited by 19
Interpretable short-term load forecasting via multi-scale temporal decomposition
Jiang, Y., Li, Y., & Chen, Y. (2024) · Electric Power Systems Research · doi:10.1016/j.epsr.2024.110781 · Cited by 17
The Impact of Recommender Type on Tourist Price Sensitivity to Travel Plans
Xu, J., Tang, X., Lin, H., & Yong, L. (2024) · Journal of Travel Research · doi:10.1177/00472875241261633 · Cited by 17
“However, people tend to believe that the decision-making process of human is easier to comprehend than that of AI (Bonezzi et al., 2022; Castelvecchi, 2016).”
Yu, B., Yan, J., Li, Y., & Xing, H. (2024) · International Journal of Disaster Risk Science · doi:10.1007/s13753-024-00578-2 · Cited by 18
Tran, T.-A., Ruppert, T., & Abonyi, J. (2024) · Computers · doi:10.3390/computers13100252 · Cited by 19
Multimodal ultrasound deep learning to detect fibrosis in early chronic kidney disease
Qin, X., Liu, X., Xia, L., Luo, Q., & Zhang, C. (2024) · Renal Failure · doi:10.1080/0886022x.2024.2417740 · Cited by 23
“But if the machine cannot explain how it makes judgments, it not only confuses doctors and patients, but also restricts its clinical application [47].”
BackgroundZhang, C., Hu, M., Wu, W., Chen, Y., Kamran, F., & Wang, X. (2025) · Teaching and Teacher Education · doi:10.1016/j.tate.2025.105086 · Cited by 19
An Immersive Node-Link Visualization of Artificial Neural Networks for Machine Learning Experts
Bellgardt, M., Scheiderer, C., & Kuhlen, T. (2020) · IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) · doi:10.1109/aivr50618.2020.00015 · Cited by 18
Chemistry-centric explanation of machine learning models
Rodríguez-Pérez, R., & Bajorath, J. (2021) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2021.100009 · Cited by 16
AI and the capitalist space economy
León, L. F. A. (2021) · Space and Polity · doi:10.1080/13562576.2021.1985852 · Cited by 17
“While humans may specify the initial goals of the artificial agents engaged in these spatial re/constructions, the specific processes and outcomes that ensue generally unfold within the black boxes characteristic of AI systems, such as neural networks (Castelvecchi, 2016).”
BackgroundSyntactic complexity in adapted extracurricular reading materials
Lei, L., & Shi, Y. (2023) · System · doi:10.1016/j.system.2023.103002 · Cited by 18
Big Data and the Study of Social Inequalities in Health: Expectations and Issues
Delpierre, C., & Kelly‐Irving, M. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00312 · Cited by 19
“A concomitantmajor challenge involves understanding and controlling the “black-box” algorithms that are used to perform many analyses and thus to produce data and results (15).”
BackgroundArtificial Intelligence: A Child's Play
Kashyap, R. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.120555 · Cited by 23
Applying knowledge-driven mechanistic inference to toxicogenomics
Tripodi, I. J., Callahan, T. J., Westfall, J., Meitzer, N. S., Dowell, R. D., & Hunter, L. (2020) · Toxicology in Vitro · doi:10.1016/j.tiv.2020.104877 · Cited by 16
Responsibility Gaps and Retributive Dispositions: Evidence from the US, Japan and Germany
Kneer, M., & Christen, M. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00509-w · Cited by 23
“Other studies, however, report similar, or higher levels of blame attribution to artificial agents than to humans across different domains (see e.g. Malle et al., 2015, 2016; Voiklis et al., 2016; Stuart & Kneer, 2021; Kneer, 2021; Liu & Du, 2022).”
BackgroundResultZhang, K., Xie, Y., He, Y., & Wang, J. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105002 · Cited by 18
Approach or avoidance? Relationship between perceived AI explainability and employee job crafting
Huo, W., Xie, J., Yan, J., Long, T., & Liang, B. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105097 · Cited by 19
Large language model-enhanced reinforcement learning for generic bus holding control strategies
Yu, J., Wang, Y., & Ma, W. (2025) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2025.104142 · Cited by 21
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.