Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 701–800 of 1,006
Beyond Automation: The Emergence of Agentic Urban AI
Tiwari, A. (2025) · Automation · doi:10.3390/automation6030029 · Cited by 22
Görmez, Y., Yağın, F. H., Yagin, B., Aygün, Y., Böke, H., Bădicu, G., Fernandes, M. S. D. S., Alkhateeb, A., Al-Rawi, M. B. A., & Aghaei, M. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1549306 · Cited by 18
“Nonetheless, a notable constraint of numerous ML models is their “black box” characteristic, rendering the decision-making process inscrutable to both healthcare personnel and patients (Rudin, 2019; Castelvecchi, 2016).”
SHAP Stability in Credit Risk Management: A Case Study in Credit Card Default Model
Lin, L., & Wang, Y. (2025) · Risks · doi:10.3390/risks13120238 · Cited by 39
“However, the complex ML architectures have raised significant concerns regarding their lack of transparency and interpretability (Castelvecchi 2016).”
Afifi, I., Elgendy, M. A., Abdelfatah, M., & El–Sappagh, S. (2025) · Brain Informatics · doi:10.1186/s40708-025-00286-7 · Cited by 17
Herrera, F., & Calderón, R. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103302 · Cited by 36
“The metaphor of the “black box” has become the dominant topic in public and academic discourse on artificial intelligence (AI) (Castelvecchi, 2016), reflecting the discomfort with systems whose internal workings resist scrutiny and accountability.”
Generative models for sequential dynamics in active inference
Parr, T., Friston, K. J., & Pezzulo, G. (2023) · Cognitive Neurodynamics · doi:10.1007/s11571-023-09963-x · Cited by 19
Stairway to heaven or highway to hell: A model for assessing cognitive automation use cases
Engel, C., Elshan, E., Ebel, P., & Leimeister, J. M. (2023) · Journal of Information Technology · doi:10.1177/02683962231185599 · Cited by 24
“Finally, the black box character refers to CA systems, particularly in the deep learning fi eld, which face challenges in explaining what happens between data input and output (Castelvecchi, 2016).”
BackgroundMethodA neural network based on the inclusion-exclusion integral and its application to data analysis
Honda, A., Itabashi, M., & James, S. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119549 · Cited by 15
NeuroXAI++: An Efficient X-AI Intensive Brain Cancer Detection and Localization
Rahman, A., Karim, M. R., Chowdhury, P., Hossain, A., & Islam, M. M. (2023) · International Conference on Next-Generation Computing, IoT and Machine Learning (NCIM) · doi:10.1109/ncim59001.2023.10212818 · Cited by 15
“Because it makes an unknown decision, which is often [17].”
BackgroundLamens, A., & Bajorath, J. (2023) · ChemMedChem · doi:10.1002/cmdc.202300586 · Cited by 18
Jiang, Y., Zhao, Q., Guan, J., Wang, Y., Chen, J., & Li, Y. (2024) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108228 · Cited by 15
Ethics and regulations for AI in radiology
Pesapane, F., & Summers, P. E. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13671-9.00001-6 · Cited by 20
The Dark Side of Powerful Platform Owners: Aspiration Adaptations of Digital Firms
Lv, D. D., & Schotter, A. P. J. (2024) · Academy of Management Perspectives · doi:10.5465/amp.2022.0169 · Cited by 23
When, What, and how should generative artificial intelligence explain to Users?
Jang, S., Lee, H., Kim, Y., Lee, D., Shin, J., & Nam, J. (2024) · Telematics and Informatics · doi:10.1016/j.tele.2024.102175 · Cited by 21
Tran, V. Q., & Byeon, H. (2024) · Digital Health · doi:10.1177/20552076241272585 · Cited by 26
Belli, O., & de Castro, H. F. (2024) · Journal of the Brazilian Society of Mechanical Sciences and Engineering · doi:10.1007/s40430-024-05222-0 · Cited by 15
“The standard AI methods present in the literature are very efficient in working with multi-dimensional data, but they lack reliability due to the black-box status [8].”
Chaotic recurrent neural networks for brain modelling: A review
Mattera, A., Alfieri, V., Granato, G., & Baldassarre, G. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107079 · Cited by 24
Patel, A., Besombes, C., Dillibabu, T., Sharma, M., Tamimi, F., Ducret, M., Chauvin, P. J., & Madathil, S. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-81724-0 · Cited by 16
Vortex memory effect of light for scattering-assisted massive data transmission
Ye, Z., Ding, C., Zhao, J., Hou, W., Men, X., Cui, Y., Liu, H., Wang, H.-B., & Xiong, J. (2025) · Optica · doi:10.1364/optica.544908 · Cited by 14
Tao, Y., Zou, W., Cremer, D., & Kraka, E. (2017) · The Journal of Physical Chemistry A · doi:10.1021/acs.jpca.7b08298 · Cited by 15
Theoretical, Measured, and Subjective Responsibility in Aided Decision Making
Douer, N., & Meyer, J. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3425732 · Cited by 22
“…artificial intelligence, machine learning, and neural networks, developers and users may be unable to fully control or predict all possible behaviors and outcomes, since their internal structure can be opaque (a “black box”) and sometimes can yield odd and counterintuitive results [17, 81].”
BackgroundA new approach to medical diagnostic decision support
Loeb, G. E. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103723 · Cited by 17
“No matter what data it is fed, its underlying algorithm is deterministic, transparent and unchanging, unlike deep-learning neural networks [34].”
BackgroundLi, F., & Lü, Y. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.136 · Cited by 19
“The black-box nature of the algorithmic process has led to focus on research in the explainability of AI, e.g., exploring the role of explainability and transparency features in adopting individualized recommendation services (Castelvecchi, 2016).”
BackgroundExplainable AI for Fighting COVID-19 Pandemic: Opportunities, Challenges, and Future Prospects
Abiodun, M. K., Awotunde, J. B., Aremu, D. R., & Adeniyi, A. E. (2021) · Disruptive technologies and digital transformations for society 5.0 · doi:10.1007/978-981-16-3783-4_15 · Cited by 21
INN: An Interpretable Neural Network for AI Incubation in Manufacturing
Chen, X., Zeng, Y., Kang, S., & Jin, R. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3519313 · Cited by 16
“The challenges are not only related to deining humans’ role in highly automated, AI-rich manufacturing environment and developing associated digital skills for humans, but also ensuring the AI systems act only in the best interest of humans, such as performance, interpretability, trustworthiness, and resilience [9].”
BackgroundImprove the Deep Learning Models in Forestry Based on Explanations and Expertise
Cheng, X., Doosthosseini, A., & Kunkel, J. M. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.902105 · Cited by 18
“Deep learning models are regarded as black boxes due to their complicated network structures and a large number of parameters (Castelvecchi, 2016).”
BackgroundIoT Network Attack Detection Using XAI and Reliability Analysis
Tabassum, S., Parvin, N., Hossain, N., Tasnim, A., Rahman, R., & Hossain, M. I. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10055236 · Cited by 15
AI-Driven Automation in a Human-Centered Cyber World
Smith, N. L., Teerawanit, J., & Hamid, O. H. (2018) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2018.00551 · Cited by 18
FinBERT—A Deep Learning Approach to Extracting Textual InformationPreprint
Huang, A., Wang, H., & Yang, Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3910214 · Cited by 14
Role of the Genomics–Phenomics–Agronomy Paradigm in Plant Breeding
Chen, C. J., Rutkoski, J. E., Schnable, J. C., Murray, S. C., Wang, L., Jin, X., Stich, B., Crossa, J. L., Hayes, B. J., & Zhang, Z. (2022) · Wiley eBooks · doi:10.1002/9781119874157.ch10 · Cited by 18
Kim, J. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113110 · Cited by 13
Digital twins and AI for healthy and sustainable cities
Birkin, M. H., Ballantyne, P., Bullock, S., Heppenstall, A. J., Kwon, H. R., Malleson, N., Yao, J., & Zanchetta, A. (2025) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2025.102305 · Cited by 17
Calderón, R., & Herrera, F. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04650-0 · Cited by 23
“Transparency, explainability, Evaluation & Auditing, controllability, opacity and black box issues Anderljung et al., 2023; Castelvecchi, 2016; Hendrycks et al., 2023; Ji et al., 2023; Mökander & Floridi, 2023; Wang et al., 2023.”
Duan, J., Zhang, M., Song, M., Xu, X., & Lu, H. (2025) · Bioengineering · doi:10.3390/bioengineering12090954 · Cited by 19
Lee, I.-H., Li, Z., & Ma, L. (2025) · Current Research in Food Science · doi:10.1016/j.crfs.2025.101196 · Cited by 21
“For example, in cancer classification using tissue images, deep learning models were shown to rely on artifacts, such as marks indicating a cancer patient, rather than the tissue itself (Castelvecchi, 2016).”
Uncertainty explanation of artificial intelligence models by SHAP
Cheng, X., Wang, T., Zhu, D., & Ma, J. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115437 · Cited by 21
Campos-Vargas, C., Sánchez-Azofeifa, A. G., Laakso, K., & Marzahn, P. (2020) · Forests · doi:10.3390/f11080827 · Cited by 13
“This outcome conforms to the argu ent by Castelvecchi [42], who suggested that ML models sh uld not be used as a black box, but instead, parametrization should be carefully considered.”
ResultDuchevet, A., Imbert, J.-P., Hogue, T. D. L., Ferreira, A., Moens, L., Colomer, A., Cantero, J., Bejarano, C., & Vázquez, A. L. R. (2022) · Transportation research procedia · doi:10.1016/j.trpro.2022.12.025 · Cited by 17
Dong, H., Wang, X., Qiu, Y., Lou, C., Ye, Y., Feng, H., Ye, X., & Chen, D. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-04854-4 · Cited by 13
“2020), but the black box DL models might prevent doctors from actually applying them in clinical practice (Castelvecchi 2016; Rudin 2019).”
BackgroundA multi-model architecture based on deep learning for aircraft load prediction
Sun, C., Li, H., Dui, H., Hong, S., Sun, Y., Song, M., Cai, D., Zhang, B., Wang, Q., Wang, Y., et al. (2023) · Communications Engineering · doi:10.1038/s44172-023-00100-4 · Cited by 14
“Please further elaborate on the applicability and limitations of the methods proposed in the article”
Matias, J. N. (2023) · Scientific Reports · doi:10.1038/s41598-023-38277-5 · Cited by 20
Global geometry of chemical graph neural network representations in terms of chemical moieties
El-Samman, A. M., Incé A. Husain, Huynh, M., Castro, S. D., Morton, B., Baerdemacker, S. D., & Baerdemacker, S. D. (2024) · Digital Discovery · doi:10.1039/d3dd00200d · Cited by 23
Multistatic-Radar RCS-Signature Recognition of Aerial Vehicles: A Bayesian Fusion Approach
Potter, M., Akçakaya, M., Necsoiu, M., Schirner, G., Deniz Erdogmus, B., & Imbiriba, T. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2024.3445323 · Cited by 20
“While numerous DL approaches have been utilized in RCS based RATR classification frameworks, challenges still persist, including the necessity for substantial data to generalize to unseen data [51], overconfident predictions [52], and a lack of interpretability [53].”
MethodSu, Y., Li, Y., Zhang, H., Yang, W., Liu, M., Luo, X., & Liu, L. (2024) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2024.108386 · Cited by 14
Artificial Intelligence in Emotion Quantification : A Prospective Overview
Liu, F. (2024) · CAAI Artificial Intelligence Research · doi:10.26599/air.2024.9150040 · Cited by 29
Explainability increases trust resilience in intelligent agents
Xu, M., & Wang, Y. (2024) · British Journal of Psychology · doi:10.1111/bjop.12740 · Cited by 18
Explainable post hoc portfolio management financial policy of a Deep Reinforcement Learning agent
Escudero, A. D. L. R., Garrido‐Merchán, E. C., & Vaca, M. C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315528 · Cited by 27
“Despite their capability as universal function approximators [10], DNNs suffer as they lack interpretability [11].”
BackgroundZhou, Y., Kong, R., Xu, Z., Xu, L., & Cheng, S. L. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2024jh000409 · Cited by 36
“However, the complexity of deep learning models often leads to a ’black box’ problem where the decision-making process is neither transparent nor understandable (Castelvecchi, 2016).”
The Impact of Learning About AI Advancements on Trust
Nikolova, M., & Angrisani, M. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.102958 · Cited by 18
Interactive imitation learning for spacecraft path-planning in binary asteroid systems
Parmar, K., & Guzzetti, D. (2021) · Advances in Space Research · doi:10.1016/j.asr.2021.04.023 · Cited by 13
Arefin, R., Samad, M. D., Akyelken, F. A., & Davanian, A. M. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00020 · Cited by 14
Do Automated Vehicles Reduce the Risk of Crashes–Dream or Reality?
Török, Á. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3212280 · Cited by 17
“Furthermore, it is also worth noting that the literature review confirms that operation under conditions outside the operational design domain (ODD) of an automated function is often identified as an important crash factor, which is also the responsibility of the driver [9].”
BackgroundContinuous diagnosis and prognosis by controlling the update process of deep neural networks
Sun, C., Li, H., Song, M., Cai, D., Zhang, B., & Hong, S. (2023) · Patterns · doi:10.1016/j.patter.2023.100687 · Cited by 16
“The field of artificial intelligence holds no consensus regarding its definition and common opinion holds that DL models are uninterpretable black-boxes [19].”
A Novel Prediction Model for Colon Cancer Recurrence Using Auto-artificial Intelligence
Mazaki, J., Katsumata, K., Ohno, Y., Udo, R., Tago, T., Kasahara, K., Kuwabara, H., Enomoto, M., Ishizaki, T., Nagakawa, Y., et al. (2021) · Anticancer Research · doi:10.21873/anticanres.15276 · Cited by 14
Binary logistic regression modeling with TensorFlow™
Zhang, Z., Mo, L., Chen, H., & Xu, P. (2019) · Annals of Translational Medicine · doi:10.21037/atm.2019.09.125 · Cited by 13
“01)) { predBi <- pred >= cutoff; crossTab <- table(predBi, dat_test$mort) accuracy = (crossTab[1]+crossTab[4])/sum(crossTab) DTaccuracy <- rbind(DTaccuracy,c(accuracy,cutoff)) } > names(DTaccuracy) <- c('Accuracy','Cutoff') > qplot(x=Cutoff, y = Accuracy, data = DTaccuracy)”
Tarhini, A. A., & Naqa, I. E. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26168002 · Cited by 13
A paradox perspective on early AI adoption: understanding temporal and relational tensions
Engström, A., Pittino, D., Mohlin, A., Mirzaei, N. E., & Johansson, A. (2025) · Journal of Organizational Change Management · doi:10.1108/jocm-02-2025-0086 · Cited by 19
Pesapane, F., Cecco, C. N. D., Wang, H., Hauglid, M. K., & Sardanelli, F. (2026) · European Radiology · doi:10.1007/s00330-026-12460-4 · Cited by 19
“Even though full transparency is unrealistic, accountability mechanisms should compensate for opacity [39, 40] through a combination of measures that can be operationally validated and audited, such as quantitative con fi dence/uncertainty readouts, structured error pro fi les (including subgroup and…”
Çevikbaş, M., & Kaiser, G. (2026) · ZDM · doi:10.1007/s11858-026-01786-4 · Cited by 12
“Without proper scaffolding, technology risks becoming a black box that undermines sense-making and learner autonomy (Castelvecchi, 2016; Siller et al., 2022).”
Application of deep-learning to the seronegative side of the NMO spectrum
Cacciaguerra, L., Storelli, L., Radaelli, M., Mesaroš, Š., Moiola, L., Drulovic, J. S., Filippi, M., & Rocca, M. A. (2021) · Journal of Neurology · doi:10.1007/s00415-021-10727-y · Cited by 13
Data-driven identification of complex disease phenotypes
Strauss, M. J., Niederkrotenthaler, T., Stefan Thurner, A., Kautzky‐Willer, A., & Klimek, P. (2021) · Journal of The Royal Society Interface · doi:10.1098/rsif.2020.1040 · Cited by 12
“These methods have yielded high predictive performance, but their transparence and interpretability are often limited [16,17].”
MethodBeyond English: Considering Language and Culture in Psychological Text Analysis
Kučera, D., & Mehl, M. R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.819543 · Cited by 20
“However, from the perspective of our paper, their disadvantage is the problematic interpretation of the analytical processes itself, i.e., the so-called black box problem (Castelvecchi, 2016).”
BackgroundHewitt, R. J., Roodposhti, M. S., & Bryan, B. A. (2022) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2022.2098299 · Cited by 12
“198, Jabbar and Khan 2015) and, perhaps most importantly, the fact that the transition rules thus derived are rarely understandable by humans – a model developed through these kinds of artificial intelligence techniques is effectively a black box able to discover and apply transition rules, but not explain them (Caste…”
BackgroundArtificial intelligence and digital twins in power systems: Trends, synergies and opportunities
Shen, Z., Arraño-Vargas, F., & Konstantinou, G. S. (2022) · Digital Twin · doi:10.12688/digitaltwin.17632.1 · Cited by 60
“• Interpretability: AI models work as black boxes; users can see the inputs and outputs but do not have a view of the process in between 60 .”
BackgroundPálfi, B., Arora, K., Prociuk, D., & Kostopoulou, O. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.108102 · Cited by 18
Zawati, M. H., & Lang, M. (2024) · Journal of Medical Internet Research · doi:10.2196/50344 · Cited by 15
“Artificially intelligent systems often face what is sometimes described as a “black box” or inscrutability problem [38].”
BackgroundMeiser, M., & Zinnikus, I. (2024) · Energies · doi:10.3390/en17091992 · Cited by 21
Mills, N., Issadeen, Z., Matharaarachchi, A., Bandaragoda, T., Silva, D. D., Jennings, A., & Manic, M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00123-6 · Cited by 20
Ethical considerations of AI in financial decision
Wang, Y. (2024) · Computing and artificial intelligence. · doi:10.59400/cai.v2i1.1290 · Cited by 27
“The lack of transparency can lead to mistrust and reluctance to adopt AI systems, despite their potential benefits [8].”
BackgroundExplainable Ensemble Learning Approaches for Predicting the Compression Index of Clays
Ge, Q., Xia, Y., Shu, J., Li, J., & Sun, H. (2024) · Journal of Marine Science and Engineering · doi:10.3390/jmse12101701 · Cited by 16
“…in real-world test data, which frequently contains missing items or values due to diverse collection meth-ods/sites, and (3) the “black box” nature of ML models [24–26] makes it difficult to discern the precise relationships between geotechnical inputs and the resulting compression index.”
Foley, K. M., McLean, C., Zylva, R. D., Asa, G. A., Maio, J., Batchelor, S., Dzando, G., & Dimassi, A. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251316249 · Cited by 16
“…generative AI which seeks to replicate human thinking) rely on the formation of thousands of intricately-devised rules developed through iterative interactions with millions or billions of training data items – conveyed linguistically (Castelvecchi, 2016; Holzinger, 2016; Wagner et al., 2022).”
Integrated Analytical Framework for the Development of Artificial Intelligence-Based Medical Devices
Arima, H., & Kano, S. (2021) · Therapeutic Innovation & Regulatory Science · doi:10.1007/s43441-021-00292-x · Cited by 16
“The black-box characteristic of deep learning and the trans-parency to provide results may lead to legal accountabili-ties and implications in AI utilization [44].”
BackgroundMachine learning in the prognostic appraisal of Class III growth
Perillo, L., Auconi, P., d’Apuzzo, F., Grassia, V., Scazzocchio, M., Nucci, L., McNamara, J. A., & Franchi, L. (2021) · Seminars in Orthodontics · doi:10.1053/j.sodo.2021.05.006 · Cited by 14
Liao, A., Chen, J.-R., Liu, S.-H., Lu, C.-H., Lin, C.-W., Shieh, J., Weng, W., & Tsui, P. (2021) · Diagnostics · doi:10.3390/diagnostics11060963 · Cited by 15
Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning
Parra-Ullauri, J. M., García‐Domínguez, A., Bencomo, N., Zheng, C., Chen, Z., Boubeta-Puig, J., Ortiz, G., & Yang, S. (2021) · Software & Systems Modeling · doi:10.1007/s10270-021-00952-4 · Cited by 17
“Despite its broad applicability, the nature of ML is still considered as a “black-box” where system decisions can become opaque to stakeholders [12].”
BackgroundIncreased Need for Data Analytics Education in Support of Verification and Validation
Lynch, C. J., Gore, R. J., Collins, A. J., Cotter, T. S., Grigoryan, G., & Leathrum, J. F. (2021) · Winter Simulation Conference · doi:10.1109/wsc52266.2021.9715485 · Cited by 14
Explainable Machine Learning for Longitudinal Multi-Omic Microbiome
Laccourreye, P., Bielza, C., & Larrañaga, P. (2022) · Mathematics · doi:10.3390/math10121994 · Cited by 12
Liu, D., Wang, W., Wang, X., Shi, Z., Sacchi, M. D., & Chen, W. (2022) · Geophysics · doi:10.1190/geo2022-0179.1 · Cited by 12
Discriminative patterns of white matter changes in Alzheimer's
Srivishagan, S., Kumaralingam, L., Thanikasalam, K., Pinidiyaarachchi, U. A. J., & Ratnarajah, N. (2022) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2022.111576 · Cited by 13
“However, the DNNs ’ decisions are frequently seen as non-transparent (Castelvecchi, 2016), making it challenging to use these algorithms in clinical practice.”
Jiang, H., Li, C., Cai, J., & Wang, J. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591753 · Cited by 18
“Deep learning-based methods [1, 4, 25, 26, 34, 45, 51– 54 , 66”
Zhang, B., Cheng, S., Zhao, Y., & Lu, F. (2023) · Sustainable Cities and Society · doi:10.1016/j.scs.2023.104834 · Cited by 12
Information-Based Medicine in Glioma Patients: A Clinical Perspective
Senders, J. T., Harary, M., Stopa, B. M., Staples, P. C., Broekman, M. L. D., Smith, T. R., Gormley, W. B., & Arnaout, O. M. (2018) · Computational and Mathematical Methods in Medicine · doi:10.1155/2018/8572058 · Cited by 12
Trinh, M. K., Wayland, M. T., & Prabakaran, S. (2019) · Journal of The Royal Society Interface · doi:10.1098/rsif.2019.0410 · Cited by 13
Explainable AI-enabled hybrid deep learning architecture for breast cancer detection
Zou, Y., & Miao, P. (2025) · Frontiers in Immunology · doi:10.3389/fimmu.2025.1658741 · Cited by 24
“The major drawback that hinders using DL models for Breast Cancer Detection is their black-box nature (17).”
A perspective on embracing emerging technologies research for organizational behavior
Philip, J. (2021) · Organization Management Journal · doi:10.1108/omj-10-2020-1063 · Cited by 14
Lee, Y., Lin, C., Fang, W., Lee, C., Ho, C., Wang, C., Tsai, D., & Lin, C. S. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.895201 · Cited by 12
“Finally, our ECG-Alb must become more transparent due to the DLM’s “black box” (41).”
MethodExplainable deep drug–target representations for binding affinity prediction
Monteiro, N. R. C., Simões, C. J. V., Ávila, H. V., Abbasi, M., Oliveira, J. L., & Arrais, J. P. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04767-y · Cited by 17
“However, they are still considered to be opaque and devoided of transparency in their inner operations and results [7, 8].”
BackgroundArtificial Intelligence Approaches in Drug Discovery: Towards the Laboratory of the Future
Frusciante, L., Visibelli, A., Geminiani, M., Santucci, A., & Spiga, O. (2022) · Current Topics in Medicinal Chemistry · doi:10.2174/1568026622666221006140825 · Cited by 16
Liu, P., Lin, C. S., Lin, C., Fang, W., Lee, C., Wang, C., & Tsai, D. (2023) · Diagnostics · doi:10.3390/diagnostics13172723 · Cited by 12
Panagoulias, D. P., Virvou, M. K., & Tsihrintzis, G. A. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00077 · Cited by 19
“This is usually referred to as the “black box paradox” [20], [21].”
BackgroundPanagoulias, D., Virvou, M., & Tsihrintzis, G. (2023) · International Conferences on Biological Information and Biomedical Engineering · doi:10.1109/bibe60311.2023.00062 · Cited by 10
Explainable Vision Transformers for Vein Biometric Recognition
Albano, R., Giusti, L., Maiorana, E., & Campisi, P. (2024) · IEEE Access · doi:10.1109/access.2024.3393558 · Cited by 14
“Nowadays, there is a growing demand for transparent systems that may empower users to comprehend why specific decisions are made, thereby fostering confidence in their fairness and unbiasedness [13], [14], [15].”
BackgroundA Teleological Approach to Information Systems Design
Fumagalli, M., Ferrario, R., & Guizzardi, G. (2024) · Minds and Machines · doi:10.1007/s11023-024-09673-1 · Cited by 12
“This may sound like an exaggeration, but it seems that we are witnessing something similar to the above-mentioned “blindness” when discussing AI black box models (Castelvecchi, 2016), whose behaviour is often opaque to the designers themselves.”
GAGE: Genetic Algorithm-Based Graph Explainer for Malware Analysis
Saqib, M., Fung, B. C. M., Charland, P., & Walenstein, A. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00179 · Cited by 17
“Moreover, DL methods have been employed, but interpretability and human understanding of model outputs are needed [8].”
MethodChun, L., Wang, D., He, L., Li, D., Fu, Z., Xue, S., Su, X., & Zhou, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-73837-3 · Cited by 11
“While ML has undergone significant and continuous progress in the past decade, most ML algorithms, including deep learning, cause the “black-box” problem 8 .”
BackgroundNie, C., Wang, K., Zhou, H., Deng, J., Chen, Z., Zhang, K., Chen, L., Huang, D., Liang, J., & Zhao, L. (2024) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.4c15835 · Cited by 11
From scientific theory to duality of predictive artificial intelligence models
Bajorath, J. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102516 · Cited by 13
Bhattacharjee, A., Kumar, A., Ojha, P. K., & Kar, S. (2025) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2025.2491669 · Cited by 13
“The black-box nature of the AI models limits clinical acceptance by impeding understanding of the decision-making processes [200].”
Comprehensive machine learning approaches for modelling the state of charge of lithium-ion batteries
Rae, M., Ottaviani, M., Capková, D., Kazda, T., Maria, L. J. S., Ryan, K. M., Passerini, S., & Singh, M. (2025) · Journal of Power Sources · doi:10.1016/j.jpowsour.2025.236929 · Cited by 14
Avoiding virtual dystopia: A design theory for emancipatory participatory immersive platforms
Tingelhoff, F., & Marga, J. J. (2025) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2025.101910 · Cited by 12
“Indeed, modern AI algorithms are shrouded within a black box (Castelvecchi, 2016; Tingelhoff et al., 2025a) that obscures the reasons behind certain decisions, making it difficult to challenge or rectify biased outcomes (Pasquale, 2015).”
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.