Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 301–400 of 1,006
Artificial intelligence for high content imaging in drug discovery
Puigvert, J. C., & Spjuth, O. (2024) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2024.102842 · Cited by 57
“The complexity and opacity of these models often make it difficult to understand the basis of their predictions, posing challenges for biological validation and hypothesis generation [39].”
BackgroundZhang, M., Zhang, M., Yang, H., Jin, Y., Zhang, X., & Liu, H. (2021) · Remote Sensing · doi:10.3390/rs13152934 · Cited by 61
Rhim, J., Lee, J., Chen, M., & Lim, A. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.632394 · Cited by 65
“As AI-based reasoning is a blackbox Castelvecchi (2016), AV moral reasoning will be challenging to fully understand, even for those who programmed them.”
BackgroundExplanatory pragmatism: a context-sensitive framework for explainable medical AI
Nyrup, R., & Robinson, D. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09632-3 · Cited by 84
“This claim is significant, since it addresses one of the major concerns about the increased reliance on AI systems in high-stakes contexts, such as the medical domain, namely that they are “opaque” (Burrell, 2016), “black boxes” (Castelvecchi, 2016), or lacking in “interpretability” or “explainability” (Watson et al.”
BackgroundChristoforaki, M., & Beyan, O. (2022) · Applied Sciences · doi:10.3390/app12094130 · Cited by 50
Artificial intelligence in multiparametric magnetic resonance imaging: A review
Li, C., Li, W., Liu, C., Zheng, H., Cai, J., & Wang, S. (2022) · Medical Physics · doi:10.1002/mp.15936 · Cited by 57
EdgeSHAPer: Bond-centric Shapley value-based explanation method for graph neural networks
Mastropietro, A., Pasculli, G., Feldmann, C., Rodríguez-Pérez, R., & Bajorath, J. (2022) · iScience · doi:10.1016/j.isci.2022.105043 · Cited by 60
“However, a characteristic feature of most MLmethods—by nomeans confined to DNNs—is their often quoted ‘‘black box’’ character (Castelvecchi, 2016), meaning that decisions of ML models remain machine-internal and are hard, if not impossible to comprehend for humans.”
Chang-Brahim, I., Koppensteiner, L. J., Beltrame, L., Bodner, G., Saranti, A., Salzinger, J., Fanta‐Jende, P., Sulzbachner, C., Bruckmüller, F., Trognitz, F., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1319938 · Cited by 63
“This is commonly referred to as the ‘ black box problem ’ (Castelvecchi, 2016), which can ultimately lead to the inability to provide users with explanations for their decisions.”
BackgroundComes, M. C., Forgia, D. L., Didonna, V., Fanizzi, A., Giotta, F., Latorre, A., Martinelli, E., Mencattini, A., Paradiso, A., Tamborra, P., et al. (2021) · Cancers · doi:10.3390/cancers13102298 · Cited by 56
“Neural networks have made great advances and continue to have great success, though the so-called explainability of their internal structures and the consequent mechanisms of transfer of information are still open problems [46].”
BackgroundXAI-AV: Explainable Artificial Intelligence for Trust Management in Autonomous Vehicles
Mankodiya, H., Obaidat, M. S., Gupta, R., & Tanwar, S. (2021) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci52664.2021.9583190 · Cited by 53
Artificial intelligence algorithms to predict Italian real estate market prices
Rampini, L., & Cecconi, F. R. (2021) · Journal of Property Investment and Finance · doi:10.1108/jpif-08-2021-0073 · Cited by 75
Judging machines: philosophical aspects of deep learning
Schubbach, A. (2019) · Synthese · doi:10.1007/s11229-019-02167-z · Cited by 58
Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors
Ceni, A., Ashwin, P., & Livi, L. (2019) · Cognitive Computation · doi:10.1007/s12559-019-09634-2 · Cited by 62
Statistically-Robust Clustering Techniques for Mapping Spatial Hotspots: A Survey
Xie, Y., Shekhar, S., & Li, Y. (2022) · ACM Computing Surveys · doi:10.1145/3487893 · Cited by 68
“While it can be risky to directly use deep learning to find hotspots due to concerns such as non-interpretability and overfitting issues [22], these models can help preprocess large volumes of visual and text information from remote sensing data (satellite or aerial imagery, LiDAR point clouds) [78, 184], social platf…”
BackgroundDetection of Rare Objects by Flow Cytometry: Imaging, Cell Sorting, and Deep Learning Approaches
Voronin, D. V., Kozlova, A. A., Verkhovskii, R. A., Ermakov, A. V., Makarkin, M. A., Inozemtseva, O. A., & Bratashov, D. N. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21072323 · Cited by 59
“Additionally, a serious obstacle using deep learning methods in diagnostics may be their interpretability [377].”
BackgroundCohen, E. B., & Gordon, I. K. (2022) · Veterinary Radiology & Ultrasound · doi:10.1111/vru.13171 · Cited by 56
“16,17 This raises the question as to whether a veterinarian is justified to make a clinical decision (e.g. euthanize a patient) based on an algorithm output (e.g. pulmonary nodules identified on a thoracic radiograph indicating metastatic neoplasia) in which the clinician does not know how the…”
BackgroundSiemers, F. M., & Bajorath, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-33215-x · Cited by 59
“Despite progress made with many applications in these and other areas, the acceptance of ML predictions in interdisciplinary research continues to be limited due to the black box character of most ML approaches 7 .”
BackgroundGriffen, E., Dossetter, A. G., Leach, A. G., & Montague, S. (2018) · Drug Discovery Today · doi:10.1016/j.drudis.2018.03.011 · Cited by 54
“This algorithmic bias is an area of significant current concern and research [74].”
BackgroundPhenomes: the current frontier in animal breeding
Miguel Pérez-enciso, A., & Steibel, J. P. (2021) · Genetics Selection Evolution · doi:10.1186/s12711-021-00618-1 · Cited by 61
“Although model interpretability is an issue [45, 46], experience shows that opening the ‘black box’ is not needed for accurate prediction [40].”
BackgroundEvaluation-oriented façade defects detection using rule-based deep learning method
Guo, J., Wang, Q., & Li, Y. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103910 · Cited by 53
Benke, K. K., Norng, S., Robinson, N. J., Chia, K., Rees, D. B., & Hopley, J. (2020) · Geoderma · doi:10.1016/j.geoderma.2020.114210 · Cited by 52
Interpretable algorithmic forensics
Garrett, B. L., & Rudin, C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301842120 · Cited by 68
“One called it as difficult to understand black box AI as it is to “understand the networks inside” the human brain (17).”
BackgroundDeep reinforcement learning for conservation decisions
Lapeyrolerie, M., Chapman, M., Norman, K., & Boettiger, C. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13954 · Cited by 62
“Since deep neural networks lack transparency (Castelvecchi 2016), can we be confident that the deep RL agent will generalize its past experience to new situations?”
BackgroundContributions and limitations of using machine learning to predict noise-induced hearing loss
Chen, F., Cao, Z., Grais, E. M., & Zhao, F. (2021) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-020-01648-w · Cited by 53
“Con-currently, due to the nature of black box, the exact relationship between inputs and outputs is hard to interpret from ML models (Castelvecchi 2016), which may prevent researchers from focusing on the specific factors that cause high risk of hearing impairment.”
BackgroundLiu, W.-T., Lin, C.-S., Tsao, T.-P., Lee, C.-C., Cheng, C.-C., Chen, J.-T., Tsai, C.-S., Lin, W.-S., & Lin, C. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.09.028 · Cited by 46
“26 To understand the AI decisions, we used class-activation mappings to display the focus of the algorithm on ECGs and CXRs (Supplemental Figs.”
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Pais, C., Miranda, A., Carrasco, J., & Shen, Z. M. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2021.105122 · Cited by 52
de Haan, Y., van den Berg, E., Goutier, N., Kruikemeier, S., & Lecheler, S. K. (2022) · Digital Journalism · doi:10.1080/21670811.2022.2027798 · Cited by 64
Semi-supervised learning in cancer diagnostics
Eckardt, J., Bornhäuser, M., Wendt, K., & Middeke, J. M. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.960984 · Cited by 59
“ML and especially deep learning has often been referred to as a ‘black box’ (45) and the path of decision making within a model is hard to interpret.”
BackgroundLuo, Z., Zuo, R., Xiong, Y., & Zhou, B. (2023) · Natural Resources Research · doi:10.1007/s11053-023-10200-9 · Cited by 48
“Despite the excellent performance of DLAs in geochemistry exploration, their nested nonlinear network structures and deep abstraction expression make such models highly opaque (Castelvecchi, 2016; Montavon et al., 2017; Samek et al., 2017; Reichstein et al., 2019).”
BackgroundCoupled Longitudinal and Lateral Control of a Vehicle using Deep Learning
Devineau, G., Polack, P., Altché, F., & Moutarde, F. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8570020 · Cited by 56
Artificial intelligence in basic and clinical neuroscience: Opportunities and ethical challenges.
Kellmeyer, P. (2019) · Neuroforum · doi:10.1515/nf-2019-0018 · Cited by 45
Predictive modeling of pressure injury risk in patients admitted to an intensive care unit
Ladios-Martin, M., Fernández-de-Maya, J., Ballesta-López, F.-J., Belso-Garzas, A., Mas-Asencio, M., & Cabañero-Martínez, M. J. (2020) · American Journal of Critical Care · doi:10.4037/ajcc2020237 · Cited by 45
A descriptive framework for the field of deep learning applications in medical images
Tian, Y., & Fu, S. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106445 · Cited by 45
Manlhiot, C., den Eynde, J. V., Kutty, S., & Ross, H. J. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.11.009 · Cited by 55
Digital twins for electro-physical, chemical, and photonic processes
Guo, Y. B., Klink, A., Bártolo, P., & Guo, W. (2023) · CIRP Annals · doi:10.1016/j.cirp.2023.05.007 · Cited by 49
Hauptman, A. I., Schelble, B. G., Duan, W., Flathmann, C., & McNeese, N. J. (2024) · Cognition Technology & Work · doi:10.1007/s10111-024-00765-7 · Cited by 67
“This issue is exacerbated by human distrust of AI systems that make decisions within a "black box" algorithm, which hides what and how the AI is processing information to make its decisions (Castelvecchi 2016).”
Lund, B. D., Mannuru, N. R., & Agbaji, D. A. (2024) · Journal of Information Science · doi:10.1177/01655515241282001 · Cited by 70
“This ‘black box’ phenomenon in AI naturally leads many to be reluctant to embrace the technology [25].”
Transdisciplinary Foundations of Geospatial Data Science
Xie, Y., Eftelioglu, E., Ali, R. Y., Tang, X., Yan, L., Doshi, R., & Shekhar, S. (2017) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi6120395 · Cited by 44
“, black box [1]) of data mining and machine learning approaches has raised major concerns and anxiety.”
BackgroundQian, X., Zhang, Q., Liu, S., Wang, Y., Chen, X., Liu, J., Yang, Y., Chen, X., Wei, Y., Xiao, Q., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06610-0 · Cited by 46
“Based on the nature of deep learning networks, the neural network is provided only with the associated classification category, without sufficient information regarding the intermediate features being used [34].”
BackgroundLee, J., Ko, H., Seo, S., & Pack, S. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3205307 · Cited by 52
“Further, it is difficult to define a suitable accuracy function (H(ωI , Dtest)) in a closed form [27].”
BackgroundLuppi, A. I., Cabral, J., Cofré, R., Mediano, P. A. M., Rosas, F. E., Qureshi, A. Y., Kuceyeski, A. F., Tagliazucchi, E., Raimondo, F., Deco, G., et al. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120162 · Cited by 47
“This desideratum can be summarised by saying that “models must not be black-boxes ” ( Castelvecchi, 2016 ; Heinrichs and Eickhoff, 2020 ).”
BackgroundMethodClassification of Potentially Hazardous Asteroids Using Supervised Quantum Machine Learning
Bhavsar, R., Jadav, N. K., Bodkhe, U., Gupta, R., Tanwar, S., Sharma, G., Bokoro, P. N., & Sharma, R. (2023) · IEEE Access · doi:10.1109/access.2023.3297498 · Cited by 55
“Moreover, these classifiers frequently lack transparency and interpretability, attributing to their characterization as ‘‘black box’’ models [32].”
BackgroundThe AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Understanding plant phenotypes in crop breeding through explainable AI
Danilevicz, M. F., Upadhyaya, S. R., Batley, J., Bennamoun, M., Bayer, P. E., & Edwards, D. (2025) · Plant Biotechnology Journal · doi:10.1111/pbi.70208 · Cited by 53
“The post hoc explanations provided by some XAI methods are often required to trace back the feature impact on the prediction, as the sheer quantity of parameters deployed by a deep learning model results in a ‘black box’ model with non-transparent decisions (Castelvecchi, 2016).”
Long short-term memory-based recurrent neural networks for nonlinear target tracking
Gao, C. J., Yan, J., Zhou, S., Chen, B., & Liu, H. (2019) · Signal Processing · doi:10.1016/j.sigpro.2019.05.027 · Cited by 48
Bertsimas, D., Dunn, J., Gibson, E., & Orfanoudaki, A. (2022) · Machine Learning · doi:10.1007/s10994-021-06117-0 · Cited by 80
“Unlike decision trees, these approaches lead to “black-box” models which are not interpretable and provide little information about how they arrive at their predictions (Samek and Müller 2019; Castelvecchi 2016).”
BackgroundA Comprehensive Survey of Machine Learning Applied to Radar Signal ProcessingPreprint
Lang, P., Fu, X., Martorella, M., Dong, J., Qin, R., Meng, X., & Xie, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13702 · Cited by 80
“Black-box property of the DNN model [596], however, demonstrates a severe challenge in practical applications of ML algorithms, such as medical image precessing domain and bank investment decision making.”
Methodde Sio, F. S. (2021) · Ethics and Information Technology · doi:10.1007/s10676-021-09609-8 · Cited by 51
“This is known in the literature as the “black box” problem (Castelvecchi, 2016).”
BackgroundHuman–Algorithm Collaboration Works Best if Humans Lead (Because it is Fair!)
Cremer, D. D., & McGuire, J. (2021) · Social Justice Research · doi:10.1007/s11211-021-00382-z · Cited by 51
A survey on deep learning for financial risk prediction
Peng, K., & Yan, G. (2021) · Quantitative Finance and Economics · doi:10.3934/qfe.2021032 · Cited by 61
“Hence, we believe this issue will be solved gradually (Castelvecchi, 2016).”
BackgroundEthics of using artificial intelligence (AI) in veterinary medicine
Coghlan, S., & Quinn, T. P. (2023) · AI & Society · doi:10.1007/s00146-023-01686-1 · Cited by 63
“Transparency broadly refers to users’ knowledge of how an AI system arrived at its prediction (Castelvecchi 2016).”
NMR‐based metabolomics of biofluids in cancer
Giskeødegård, G. F., Madssen, T. S., Euceda, L. R., Tessem, M., Moestue, S. A., & Bathen, T. F. (2018) · NMR in Biomedicine · doi:10.1002/nbm.3927 · Cited by 48
“However, such approaches pose important challenges, including possibly stagnating the generation of biomedical knowledge, and not knowing whether the model bases its decisions on statistically associated “red herrings”, as opposed to relevant clinical factors.(119) While standardization of the NMR experiment itself is…”
Backgroundvan der Ven, W. H., Veelo, D. P., Wijnberge, M., van der Ster, B. J. P., Vlaar, A. P. J., & Geerts, B. F. (2020) · Surgery · doi:10.1016/j.surg.2020.09.041 · Cited by 48
Why Does Explainability Matter in News Analytic Systems? Proposing Explainable Analytic Journalism
Shin, D. (2021) · Journalism Studies · doi:10.1080/1461670x.2021.1916984 · Cited by 51
“Because of this relation, in AJ as well as AI, there are mounting requests for legitimate justi fi cations on why and how results are derived through existing systems (Castelvecchi 2016; Crain 2018).”
BackgroundOn the person-based predictive policing of AI
Hung, T.-W., & Yen, C.-P. (2020) · Ethics and Information Technology · doi:10.1007/s10676-020-09539-x · Cited by 68
“The way AI systems generate results is often described as opaque and inexplicable (Castelvecchi 2016; Wachter et al. 2017).”
BackgroundLi, H., Kim, J. T., Kim, J.-S., Choi, D., & Lee, S. (2023) · ACS Photonics · doi:10.1021/acsphotonics.3c00057 · Cited by 41
Yang, F., Zuo, R., Xiong, Y., Wang, J., & Zhang, G. (2023) · Journal of Geochemical Exploration · doi:10.1016/j.gexplo.2023.107274 · Cited by 45
Can Autism Be Diagnosed with Artificial Intelligence? A Narrative Review
Chaddad, A., Li, J., Lu, Q., Li, Y., Okuwobi, I. P., Tanougast, C., Desrosiers, C., & Niazi, T. M. (2021) · Diagnostics · doi:10.3390/diagnostics11112032 · Cited by 50
Wang, Y., Yang, Y., Cao, G., Guo, J., Wei, P., Feng, T., Dai, Y. D., Huang, J., Kang, G., & Zhao, G. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105703 · Cited by 48
Schweitzer, S., & Cremer, D. D. (2023) · Academy of Management Discoveries · doi:10.5465/amd.2022.0115 · Cited by 53
Bhattacharya, M., Pal, S., Chatterjee, S., Lee, S., & Chakraborty, C. (2024) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2024.102255 · Cited by 54
“38 The Black Box can be applicable to LLM or ChatGPT models.”
MethodC-SHAP: A Hybrid Method for Fast and Efficient Interpretability
Ranjbaran, G., Recupero, D. R., Roy, C. K., & Schneider, K. A. (2025) · Applied Sciences · doi:10.3390/app15020672 · Cited by 54
Luo, X., Tu, X., Ding, Y., Gao, G., & Deng, M. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz768 · Cited by 46
“Nevertheless, despite its great achievements, deep learning is also blamed for its lack of interpretability(Zou et al., 2018; Castelvecchi, 2016).”
BackgroundLin, C.-S., Lee, Y.-T., Fang, W.-H., Lou, Y.-S., Kuo, F.-C., Lee, C.-C., & Lin, C. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11080725 · Cited by 39
“Finally, the “black box” of DLM necessitates our ECG-HbA1c being more transparent [58].”
BackgroundPan, W., Wang, X., Sun, Y. L., Wang, J., Li, Y., & Li, S. (2023) · Plant Methods · doi:10.1186/s13007-023-00982-7 · Cited by 47
“Machine learning models have complex parameters and structures [104], so we plan to try a wider variety of machine learning meth-ods in the future.”
BackgroundAn explainable two-stage machine learning approach for precipitation forecast
Şenocak, A. U. G., Yılmaz, M. T., Kalkan, S., Yücel, İ., & Amjad, M. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.130375 · Cited by 38
Yang, J., Eyre, D. W., Lü, L., & Clifton, D. A. (2023) · npj Antimicrobials and Resistance · doi:10.1038/s44259-023-00015-2 · Cited by 45
“Furthermore, a DT is highly interpretable, whereas a deep neural network is less straightforward to interpret, even commonly being referred to as a “ black box ” 9 .”
Noorchenarboo, M., & Grolinger, K. (2025) · Energy and Buildings · doi:10.1016/j.enbuild.2024.115177 · Cited by 38
“Modern AI models such as Deep Neural Networks (DNNs) are complex systems with many parameters, making them dif-ficult to interpret [40].”
Ekundayo, O., & Ezugwu, A. E.-S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113378 · Cited by 62
Pareek, A., Parkes, C. W., Bernard, C. D., Abdel, M. P., Saris, D. B. F., & Krych, A. J. (2019) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-019-05792-w · Cited by 38
Wang, L., Kawaguchi, K., & Wang, P. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103230 · Cited by 39
Unbiased identification of novel subclinical imaging biomarkers using unsupervised deep learning
Waldstein, S. M., Seeböck, P., Donner, R., Sadeghipour, A., Bogunović, H., Osborne, A., & Schmidt-Erfurth, U. (2020) · Scientific Reports · doi:10.1038/s41598-020-69814-1 · Cited by 38
Lee, K.-C., Lee, K., Kang, C. H., Ahn, K., Chung, L. Y., Lee, J.-J., Hong, S., Kim, B. H., & Shim, E. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2020.1468 · Cited by 38
“Finally, the black box nature of CNNs [25,26], we thought, could be partially resolved by integrating two different methods (GP and TW) within the medical domain during the model development process.”
MethodEthical dilemmas posed by the rise of artificial intelligence: a view from transhumanism
Muriel, F. A. Z., Zapata, S. M., & Montoya-Zapata, D. (2024) · Región Científica · doi:10.58763/rc2024225 · Cited by 46
Rezazadeh, A., Jafarian, Y., & Valeshabad, A. K. (2022) · Forecasting · doi:10.3390/forecast4010015 · Cited by 46
Ridley, M. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24889 · Cited by 74
““ Opening the black box, ” making complex, opaque AI systems understandable, is XAI's central motivation (Castelvecchi, 2016).”
Rajpura, P., Cecotti, H., & Meena, Y. K. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad6593 · Cited by 58
“This consideration is influenced by Sturm et al (2016) in 2016 and the development of the first successful large-scale model like EEGNet (Lawhern et al 2018) in 2018.”
Using Conceptual Modeling to Support Machine Learning
Lukyanenko, R., Castellanos, A., Parsons, J., Tremblay, M. C., & Storey, V. C. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-21297-1_15 · Cited by 37
Genome-scale transcriptional dynamics and environmental biosensing
Graham, G., Csicsery, N., Stasiowski, E., Thouvenin, G., Mather, W., Ferry, M., Cookson, S., & Hasty, J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1913003117 · Cited by 42
“At present, a major obstacle to making scientific conclusions from machine-learning results is the black box problem: As an algorithm’s ability to model complex phenomena grows, its decision-making processes become more and more obscured from its operators (32).”
BackgroundFritz, A., Brandt, W., Gimpel, H., & Bayer, S. (2020) · De Ethica · doi:10.3384/de-ethica.2001-8819.20613 · Cited by 37
Machine learning–based prediction of health outcomes in pediatric organ transplantation recipients
Killian, M., Payrovnaziri, S. N., Gupta, D., Desai, D. M., & He, Z. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab008 · Cited by 43
“DL models are often referred to as black-box implying the fact that their underlying mechanisms are not easy to understand by the end user [35].”
BackgroundRodríguez-Pérez, R., & Bajorath, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-93771-y · Cited by 44
Artificial Intelligence in Cardiovascular Imaging: “Unexplainable” Legal and Ethical Challenges?
Lang, M., Bernier, A., & Knoppers, B. M. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.10.009 · Cited by 37
Rosemann, A., & Zhang, X. (2021) · Intelligent Medicine · doi:10.1016/j.imed.2021.12.002 · Cited by 49
Predicting Harmful Algal Blooms Using Explainable Deep Learning Models: A Comparative Study
Demiray, B. Z., Mermer, O., Baydaroğlu, Ö., & Demir, İ. (2025) · Water · doi:10.3390/w17050676 · Cited by 52
“While early AI systems were relatively interpretable, the rise of deep learning has introduced models with an enormous number of parameters, rendering their outputs difficult to explain—a concept known as black-box properties (Castelvecchi, 2016).”
BackgroundFully interpretable deep learning model of transcriptional control
Liu, Y., Barr, K., & Reinitz, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa506 · Cited by 35
“It is our hope that this example will provide insights into the interpretabilities of DNNs in general, a problem that has received wide attention in the community [12, 8, 7, 26, 50, 31, 51].”
BackgroundMind the gap! On the future of AI research
Dahlin, E. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00750-9 · Cited by 54
“It is therefore not surprising that many systems are still black-boxed to their intended users (Castelvecchi, 2016).”
BackgroundXiang, J., Shi, M., Fiala, M. A., Gao, F., Rettig, M. P., Uy, G. L., Schroeder, M. A., Weilbaecher, K. N., Stockerl-Goldstein, K. E., Mollah, S. A., et al. (2021) · Blood Advances · doi:10.1182/bloodadvances.2021005149 · Cited by 36
“Less-than-optimal, 2-5 3 10(6)/kg 36 (7-144) Less-than-optimal (20-40) 3.”
Jia, L.-L., Zhao, J.-X., Zhao, L., Tian, J., & Huang, G. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110640 · Cited by 42
Wulff, K., & Finnestrand, H. O. (2023) · AI & Society · doi:10.1007/s00146-023-01633-0 · Cited by 45
“These are called white-box learning models (Hall 2018), as opposed to ML models described as black boxes (Buhrmester et al. 2021; Castelvecchi 2016; Faraj et al. 2018).”
Wang, D.-Y., Liu, S.-G., Ding, J., Sun, A.-L., Jiang, D., Jiang, J., Zhao, J.-Z., Chen, D.-S., Ji, G., Li, N., et al. (2024) · Arthroscopy: The Journal of Arthroscopic & Related Surgery · doi:10.1016/j.arthro.2023.08.010 · Cited by 35
Extending application of explainable artificial intelligence for managers in financial organizations
Sabharwal, R., Miah, S. J., Wamba, S. F., & Cook, P. G. (2024) · Annals of Operations Research · doi:10.1007/s10479-024-05825-9 · Cited by 60
“The idea of explainable AI articulates a position which has the opposite role of AI as a “black box”, as the machine of which “its designers cannot explain why an AI arrived at a specific decision” (Castelvecchi, 2016; Sample, 2017).”
Masialeti, M., Talaei‐Khoei, A., & Yang, A. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102779 · Cited by 51
Geologically Constrained Convolutional Neural Network for Mineral Prospectivity Mapping
Yang, F., & Zuo, R. (2024) · Mathematical Geosciences · doi:10.1007/s11004-024-10141-w · Cited by 45
“The inherent black-box properties of DLAs have led to a crisis of trust among geologists regarding opaque model predictions (Castelvecchi 2016; Zuo et al. 2019).”
Caterson, J., Lewin, A., & Williamson, E. (2024) · Digital Health · doi:10.1177/20552076241272657 · Cited by 58
Chen, S., Liu, J., Yuan, H., Tao, Y., Xu, F., & Yang, L. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.125116 · Cited by 41
What Does It Mean to Learn in Deep Networks? And, How Does One Detect Adversarial Attacks?
Corneanu, C. A., Madadi, M., Escalera, S., & Martínez, A. M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00489 · Cited by 50
“This so called “black-box” problem has resulted in a lack of trust [5, 35].”
BackgroundKordmahalleh, M. M., Gorji, M., Harrison, S. H., & Homaifar, A. (2017) · BioData Mining · doi:10.1186/s13040-017-0146-4 · Cited by 35
Predictive analytics in the era of big data: opportunities and challenges
Zhang, Z. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2019.10.97 · Cited by 36
“, these models are considered to be black box algorithm because domain experts cannot easily understand how the predictors/features influence the outcome/label) (11,12).”
Background
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.