Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 101–200 of 1,006
Harnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 397
“Although the black box is not entirely untrustworthy [16], the results of formal study show that all participants demanded further clarification from ChatGPT to enhance the transparency of the results.”
Wang, Y. (2021) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2020.100872 · Cited by 162
Fairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161
“The black-box nature of AI models makes it hard for users to understand why a decision is made or how the data are processed for the decision-making [4–6].”
BackgroundDeep neural network battery life and voltage prediction by using data of one cycle only
Hsu, C.-W., Xiong, R., Chen, N.-Y., Li, J., & Tsou, N.-T. (2022) · Applied Energy · doi:10.1016/j.apenergy.2021.118134 · Cited by 158
Assessing public value failure in government adoption of artificial intelligence
Schiff, D. S., Schiff, K. J., & Pierson, P. (2022) · Public Administration · doi:10.1111/padm.12742 · Cited by 155
Pyun, K. R., Kwon, K., Yoo, M. J., Kim, K. K., Gong, D., Yeo, W., Han, S., & Ko, S. H. (2023) · National Science Review · doi:10.1093/nsr/nwad298 · Cited by 179
“Finally, the decision-making processes and the features derived from CNNs can be complex and difficult to decipher, earning them the ‘black box’ moniker due to their lack of interpretability [111 ].”
Tracing the evolution of AI in the past decade and forecasting the emerging trends
Shao, Z., Zhao, R., Yuan, S., Ding, M., & Wang, Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118221 · Cited by 153
Eling, M., Nuessle, D., & Staubli, J. (2022) · The Geneva Papers on Risk and Insurance Issues and Practice · doi:10.1057/s41288-020-00201-7 · Cited by 156
“…has greatly improved, the networks’ internal logic often remains inexplicable and incomprehensible due to their inherent complexity (Knight 2017; Castelvecchi 2016).27 Most of the discussions among insurance practitioners with regard to applying artificial intelligence for parts of their value…”
BackgroundUddin, M. Z., & Soylu, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-95947-y · Cited by 147
Artificial intelligence in psychiatry research, diagnosis, and therapy
Sun, J., Dong, Q., Wang, S.-W., Zheng, Y., Liu, X., Lu, T., Yuan, K., Shi, J., Hu, B., Hu, B., et al. (2023) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2023.103705 · Cited by 199
Disability, fairness, and algorithmic bias in AI recruitment
Tilmes, N. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09633-2 · Cited by 257
“4 While data scientists can track the inputs and outputs of complex machine learning AI systems, they cannot do the same for its internal data processing operations, which occur in a black box (Castelvecchi, 2016).”
BackgroundHassan, M. R., Islam, M. F., Uddin, M. Z., Ghoshal, G., Hassan, M. M., Huda, S., & Fortino, G. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.09.030 · Cited by 144
Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior
Camara, F., Bellotto, N., Coşar, S., Weber, F., Nathanael, D., Althoff, M., Wu, J., Ruenz, J., Dietrich, A., Markkula, G. M., et al. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3006767 · Cited by 187
Explaining deep neural networks: A survey on the global interpretation methods
Saleem, R., Yuan, B., Kurugöllü, F., Anjum, A., & Liu, L. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.129 · Cited by 219
Dzobo, K., Adotey, S. K., Thomford, N. E., & Dzobo, W. (2019) · OMICS A Journal of Integrative Biology · doi:10.1089/omi.2019.0038 · Cited by 174
“One problem when using AI, especially deep neural networks, is that it might not be possible to know how the algorithm determines the final result (Castelvecchi, 2016).”
New explainability method for BERT-based model in fake news detection
Szczepański, M., Pawlicki, M., Kozik, R., & Choraś, M. (2021) · Scientific Reports · doi:10.1038/s41598-021-03100-6 · Cited by 196
“Those come from the fact that most of the AI solutions employed, especially Deep Neural Networks (DNNs), are ’black-box’ type models 36,37 .”
BackgroundChenais, G., Lagarde, E., & Gil-Jardiné, C. (2023) · Journal of Medical Internet Research · doi:10.2196/40031 · Cited by 142
“…in the Actual and Possible Applications of AI for Emergency Services section are based on NLP, and concerns regarding the biases introduced by the growing use of large language models (ie, Bidirectional Encoder Representations from Transformers, Generative Pretrained Transformer 2, and XLNET)…”
Generative AI in higher education and beyond
Hashmi, N., & Bal, A. S. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.05.005 · Cited by 141
Challenges with developing and deploying AI models and applications in industrial systems
Sinha, S. R., & Lee, Y. M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00151-2 · Cited by 243
“While they have the capacity to solve complex problems, this often comes at the cost of transparency [45].”
Lee, J. H., Joo, I., Kang, T. W., Paik, Y. H., Sinn, D. H., Ha, S. Y., Kim, K., Choi, C., Lee, G., Yi, J., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06407-1 · Cited by 144
Jin, E. H., Lee, D., Bae, J. H., Kang, H. Y., Kwak, M.-S., Seo, J. Y., Yang, J. I., Yang, S. Y., Lim, S. H., Yim, J. Y., et al. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.02.036 · Cited by 140
Protein- and Peptide-Based Biosensors in Artificial Olfaction
Barbosa, A. J. M., Oliveira, A. R., & Roque, A. C. A. (2018) · Trends in biotechnology · doi:10.1016/j.tibtech.2018.07.004 · Cited by 142
“The possibility of mimicking nature, building artificial intelligent systems to perform complex tasks, and dealing with large sets of data is gaining increasing relevance in all areas, including biological sciences [1].”
BackgroundSocial Media, E‐Health, and Medical Ethics
Terrasse, M., Gorin, M., & Sisti, D. (2019) · Hastings Center Report · doi:10.1002/hast.975 · Cited by 136
Making the Case for Safety of Machine Learning in Highly Automated Driving
Burton, S., Gauerhof, L., & Heinzemann, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66284-8_1 · Cited by 174
Histopathology images predict multi-omics aberrations and prognoses in colorectal cancer patients
Tsai, P.-C., Lee, T.-H., Kuo, K.-C., Su, F.-Y., Lee, T.-L. M., Marostica, E., Ugai, T., Zhao, M., Lau, M. C., Väyrynen, J. P., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37179-4 · Cited by 136
“The authors sometimes describe MOMA as a platform in the manuscript, however, I cannot find a web service or software/portal for potential MOMA users based on code availability.”
The explainability paradox: Challenges for xAI in digital pathology
Evans, T., Retzlaff, C. O., Geißler, C., Kargl, M., Plass, M., Müller, H., Kiehl, T.-R., Zerbe, N., & Holzinger, A. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.03.009 · Cited by 135
Artificial intelligence in the diagnosis and management of arrhythmias
Nagarajan, V. D., Su-Lin, L., Robertus, J. L., Nienaber, C. A., Trayanova, N. A., & Ernst, S. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab544 · Cited by 150
Threats of Adversarial Attacks in DNN-Based Modulation Recognition
Lin, Y., Zhao, H., Tu, Y., Mao, S., & Dou, Z. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155389 · Cited by 155
Governance of automated image analysis and artificial intelligence analytics in healthcare
Ho, C. W. L., Soon, D., Caals, K., & Kapur, J. (2019) · Clinical Radiology · doi:10.1016/j.crad.2019.02.005 · Cited by 132
“21,67,68 Large datasets that are representative of the scope and variety of patient types and disease conditions, and available in machine-readable form, are needed to train and test AI algorithms, along with complete and well-structured metadata (including information on data sources).”
BackgroundInterdisciplinary Research in Artificial Intelligence: Challenges and Opportunities
Kusters, R., Misevic, D., Berry, H., Cully, A., Le Cunff, Y., Dandoy, L., Díaz-Rodríguez, N., Ficher, M., Grizou, J., Othmani, A., et al. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.577974 · Cited by 131
“Opening the black box has already been set as an important goal in AI research (Castelvecchi, 2016), even if not all experts agree that this is necessary (Holm, 2019).”
BackgroundLambers, K., Verschoof-van der Vaart, W., & Bourgeois, Q. (2019) · Remote Sensing · doi:10.3390/rs11070794 · Cited by 130
“While this has led to discussions about the ‘black box problem’ [55], the favorable results of CNN applications have outweighed most skeptical voices.”
Automated Dermatological Diagnosis: Hype or Reality?
Navarrete-Dechent, C., Dusza, S. W., Liopyris, K., Marghoob, A. A., Halpern, A. C., & Marchetti, M. A. (2018) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2018.04.040 · Cited by 129
Gallo, S., El-Gazzar, A., Zhutovsky, P., Thomas, R. M., Javaheripour, N., Li, M., Bartova, L., Bathula, D. R., Dannlowski, U., Davey, C. G., et al. (2023) · Molecular Psychiatry · doi:10.1038/s41380-023-01977-5 · Cited by 160
“All study participants provided written informed consent at their local institution.”
BackgroundTaj, I., & Jhanjhi, N. Z. (2022) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/120128 · Cited by 145
“The black box problem [81] makes it impossible to test all the unpredictable road scenarios that an autonomous vehicle”
BackgroundMethodThe growing ubiquity of algorithms in society: implications, impacts and innovations
Olhede, S. C., & Wolfe, P. J. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0364 · Cited by 120
Li, Y., Zeng, H., Zhang, M., Wu, B., Zhao, Y., Yao, X., Cheng, T., Qin, X., & Wu, F. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103269 · Cited by 172
Artificial Intelligence in Skin Cancer Diagnosis: A Reality Check
Brancaccio, G., Balato, A., Malvehy, J., Puig, S., Argenziano, G., & Kittler, H. (2023) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2023.10.004 · Cited by 159
AI explainability and governance in smart energy systems: A review
Alsaigh, R., Mehmood, R., & Katib, I. (2023) · Frontiers in Energy Research · doi:10.3389/fenrg.2023.1071291 · Cited by 121
Trust in artificial intelligence for medical diagnoses
(2020) · Progress in Brain Research · doi:10.1016/bs.pbr.2020.06.006 · Cited by 116
“This reduced interest in how doctors make diagnoses and decide on treatments is good news for AI, considering its complexity and that the computations leading to a certain outcome are often impenetrable to humans (e.g., the black box problem, see Castelvecchi, 2016).”
BackgroundBayesian networks for interpretable machine learning and optimization
Mihaljević, B., Bielza, C., & Larrañaga, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.138 · Cited by 123
“Many of the machine learning models used are black boxes [2] that do not explain their predictions in a way that a human can understand.”
BackgroundThe black box problem revisited. Real and imaginary challenges for automated legal decision making
Brożek, B., Furman, M., Jakubiec, M., & Kucharzyk, B. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09356-9 · Cited by 115
“One of the most pressing problems related to the use of AI in the decision-mak - ing processes is the so-called black box problem (Castelvecchi 2016 ; Rudin 2019 ).”
BackgroundDeep Neural Networks for Classification of LC-MS Spectral Peaks
Kantz, E. D., Tiwari, S., Watrous, J. D., Cheng, S. C., & Jain, M. (2019) · Analytical Chemistry · doi:10.1021/acs.analchem.9b02983 · Cited by 125
MedGA: A novel evolutionary method for image enhancement in medical imaging systems
Rundo, L., Tangherloni, A., Nobile, M. S., Militello, C., Besozzi, D., Mauri, G., & Cazzaniga, P. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.013 · Cited by 130
Machine Learning and Artificial Intelligence in the Multi-Omics Approach to Gut Microbiota
Rozera, T., Pasolli, E., Segata, N., & Ianiro, G. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.02.035 · Cited by 112
DeepAISE – An interpretable and recurrent neural survival model for early prediction of sepsis
Shashikumar, S. P., Josef, C. S., Sharma, A., & Nemati, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102036 · Cited by 109
“A major barrier to wide adoption of modern machine learning based CDS tools in clinical practice has been their “black box” nature [38].”
BackgroundBlack Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security
Kuppa, A., & Le-Khac, N.-A. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206780 · Cited by 123
“Not being able to understand what has been captured by the model [6] and stored in its internal representation may entail a privacy breach.”
BackgroundSemantic Communication: A Survey on Research Landscape, Challenges, and Future Directions
Getu, T. M., Kaddoum, G., & Bennis, M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3520707 · Cited by 110
Lin, C.-S., Lin, C., Fang, W.-H., Hsu, C.-J., Chen, S.-J., Huang, K.-H., Lin, W.-S., Tsai, C.-S., Kuo, C.-C., Chau, T., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15931 · Cited by 109
Food fraud detection using explainable artificial intelligence
Buyuktepe, O., Catal, C., Kar, G., Bouzembrak, Y., Marvin, H. J. P., & Gavai, A. (2023) · Expert Systems · doi:10.1111/exsy.13387 · Cited by 133
“DNNs are criticized for being black-box models because they have millions of parameters and multilayer nonlinear architectures (Castelvecchi, 2016).”
netDx: interpretable patient classification using integrated patient similarity networks
Pai, S., Hui, S., Isserlin, R., Shah, M. A., Kaka, H., & Bader, G. D. (2019) · Molecular Systems Biology · doi:10.15252/msb.20188497 · Cited by 103
“Black box methods are unlikely to be clinically successful, as physicians frequently must understand the characteristic features of a disease to make a confident diagnosis (Castelvecchi, 2016).”
BackgroundDong, L., Yang, Q., Zhang, R., & Wei, W. B. (2021) · EClinicalMedicine · doi:10.1016/j.eclinm.2021.100875 · Cited by 114
“Furthermore, since AI is a “black box”, [49] it may affect the perception and acceptance of AI in further applications.”
BackgroundMursch‐Edlmayr, A. S., Ng, W. S., Diniz‐Filho, A., Sousa, D. C., Arnould, L., Schlenker, M. B., Dueñas-Ángeles, K., Keane, P. A., Crowston, J. G., & Jayaram, H. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.55 · Cited by 123
Machine learning and artificial intelligence in neuroscience: A primer for researchers
Badrulhisham, F., Pogatzki-Zahn, E. M., Segelcke, D., Spisák, T., & Vollert, J. (2023) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2023.11.005 · Cited by 126
“In contrast to the black box of classic AI, they aim to create a white box, where decision making can be traced and understood (Castelvecchi, 2016).”
Lee, H. H., & Kim, H. (2019) · Magnetic Resonance in Medicine · doi:10.1002/mrm.27727 · Cited by 115
“46 For those who may be interested, the layer activations of the CNN are shown in Supporting Information 5 (Supporting Information Figures S5 through S8) for several representative network layers.”
BackgroundImproving plankton image classification using context metadata
Ellen, J. S., Graff, C. A., & Ohman, M. D. (2019) · Limnology and Oceanography Methods · doi:10.1002/lom3.10324 · Cited by 120
“In a CNN, the fi rst layer of fi lter weights can be rendered, but the interaction architecture of a CNN causes subsequent layers to lack a straightforward visualization, although this is an area of open research (Castelvecchi 2016).”
BackgroundHuman–AI adaptive dynamics drives the emergence of information cocoons
Piao, J., Liu, J., Zhang, F., Su, J., & Li, Y. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00731-4 · Cited by 137
Yanagihara, R. T., Lee, C. S., Ting, D. S. W., & Lee, A. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.11 · Cited by 120
A Survey on XAI for 5G and Beyond Security: Technical Aspects, Challenges and Research Directions
Senevirathna, T., La, V. H., Marchal, S., Siniarski, B., Liyanage, M., & Wang, S. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3437248 · Cited by 148
“DNNs are considered sophisticated black-box models since they have hundreds of layers and millions of parameters [53].”
Your evidence? Machine learning algorithms for medical diagnosis and prediction
Heinrichs, B., & Eickhoff, S. B. (2020) · Human Brain Mapping · doi:10.1002/hbm.24886 · Cited by 99
“The reason for this above mentioned predicament is that most current machine learning algorithms give rise to what has been called “ the black-box problem ” (Castelvecchi, 2016).”
BackgroundA Review of Solar Forecasting Techniques and the Role of Artificial Intelligence
Barhmi, K., Heynen, C., Golroodbari, S., & van Sark, W. G. J. H. M. (2024) · Solar · doi:10.3390/solar4010005 · Cited by 140
“This is more of a general issue in using AI, prompting new developments such as explainable AI (XAI) [93].”
BackgroundAn introduction to machine learning for classification and prediction
Black, J. E., Kueper, J. K., & Williamson, T. S. (2022) · Family Practice · doi:10.1093/fampra/cmac104 · Cited by 122
Ochmann, J., Michels, L., Tiefenbeck, V., Maier, C., & Laumer, S. (2024) · Information Systems Journal · doi:10.1111/isj.12482 · Cited by 95
“For many years, one of the key characteristics of highly sophisticated algorithms in decision-making was their opaqueness or ‘ black box ’ nature, referring to an incomprehensibility of how their decision was reached (Adadi & Berrada, 2018; Barredo Arrieta et al., 2020; Castelvecchi, 2016).”
BackgroundPatel, S. K., George, B., & Rai, V. (2020) · Frontiers in Pharmacology · doi:10.3389/fphar.2020.01177 · Cited by 109
“This could be achieved at different levels of data processing and abstraction, however the development of methods for interpretingMLmodels is at a relatively early stage, particularly for precision oncology (Castelvecchi, 2016).”
MethodReimagining Healthcare: Unleashing the Power of Artificial Intelligence in Medicine
Iqbal, J., Cortés Jaimes, D. C., Makineni, P., Subramani, S., Hemaida, S., Thugu, T. R., Butt, A. N., Sikto, J. T., Kaur, P., Lak, M. A., et al. (2023) · Cureus · doi:10.7759/cureus.44658 · Cited by 94
“The use of AI is not without its challenges, such as black box problems [128], data privacy, and liability issues [129].”
BackgroundHow to certify machine learning based safety-critical systems? A systematic literature review
Tambon, F., Laberge, G., An, L., Nikanjam, A., Mindom, P. S. N., Pequignot, Y., Khomh, F., Antoniol, G., Merlo, E., & Laviolette, F. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00337-x · Cited by 157
“To include more potential papers, we also used black box (deep learning models are often considered as non-interpretable black boxes [31, 205], so certification approaches for such black boxes can be useful), supervised and unsupervised (because people may use the term “(un)supervised learning” to refer to their machi…”
BackgroundReinforcement Learning for Precision Oncology
Eckardt, J., Wendt, K., Bornhäuser, M., & Middeke, J. M. (2021) · Cancers · doi:10.3390/cancers13184624 · Cited by 113
Arizmendi, C. J., Bernacki, M. L., Raković, M., Plumley, R. D., Urban, C. J., Panter, A. T., Greene, J. A., & Gates, K. M. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01939-9 · Cited by 165
“Model interpretation, however, is difficult, and is often referred to as a ”black box” (Castelvecchi, 2016) because model parameters are not inherently interpretable due to overparameterization.”
BackgroundKalakoti, R., Bahsi, H., & Nõmm, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3360626 · Cited by 93
“Therefore, DNNs are considered sophisticated black-box models because they contain thousands of layers and millions of parameters (weights), so the behavior of these models cannot be understood [33].”
BackgroundWindisch, P. Y., Weber, P., Fürweger, C., Ehret, F., Kufeld, M., Zwahlen, D. R., & Muacevic, A. (2020) · Neuroradiology · doi:10.1007/s00234-020-02465-1 · Cited by 94
Zhang, J., Zhong, F., He, K., Ji, M., Li, S., & Li, C. (2023) · Diagnostics · doi:10.3390/diagnostics13233506 · Cited by 132
“Humans cannot really understand how a machine works, even if it is actually inspired by humans [136].”
BackgroundKim, S.-W., & Lee, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12172-y · Cited by 88
“Unlike computers and robots, AI is an intangible algorithm that does not have a form; indeed, it is like a black box (Castelvecchi, 2016; Milne & Rowe, 2002; Tan et al., 2009).”
A review of explainable artificial intelligence in smart manufacturing
Madathil, A. P., Luo, X., Liu, Q., Walker, C. T., Madarkar, R., & Qin, Y. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2513574 · Cited by 146
Illimoottil, M., & Ginat, D. T. (2023) · Cancers · doi:10.3390/cancers15133267 · Cited by 110
“Deep learning models, particularly CNNs, are often considered “black boxes” due to their complex architectures and the lack of transparency in the decision-making process [47].”
BackgroundSmith, K., Piccinini, F., Balassa, T., Koos, K., Danka, T., Azizpour, H., & Horvath, P. (2018) · Cell Systems · doi:10.1016/j.cels.2018.06.001 · Cited by 87
Graph Representation Forecasting of Patient's Medical Conditions: Toward a Digital Twin
Barbiero, P., Viñas, R., & Lió, P. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.652907 · Cited by 110
“Such models exhibit great capacity (Hornik, 1991) but understanding their behavior and following their decision-making process is not trivial (Castelvecchi, 2016).”
BackgroundTaj, I., & Zaman, N. (2022) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/120124 · Cited by 152
Keel, S., Li, Z., Scheetz, J., Robman, L., Phung, J., Makeyeva, G., Aung, K., Liu, C., Yan, X., Meng, W., et al. (2019) · Clinical & Experimental Ophthalmology · doi:10.1111/ceo.13575 · Cited by 85
“differences including camera setup illumination and inclusion criteria.(21) In comparison, our DLA was trained using images from multiple clinical settings using various retinal cameras and imaging protocols, and was able to achieve excellent diagnostic accuracy.”
MethodGevaert, A. B., Adams, V., Bahls, M., Bowen, T. S., Cornelissen, V., Dörr, M., Hansen, D., Kemps, H., Leeson, P., Craenenbroeck, E. M. V., et al. (2019) · European Journal of Preventive Cardiology · doi:10.1177/2047487319877716 · Cited by 90
“More complex machine learning processes, such as neural networks, build layer upon layer of automated decisions up to a point where it is impossible to retrace the individual steps.(148) Thus, while some neural networks have been proven to outperform humans (for example in image recognition(149)), it is often hard to…”
BackgroundFacilitating topic modeling in tourism research:Comprehensive comparison of new AI technologies
Kirilenko, A., & Stepchenkova, S. O. (2024) · Tourism Management · doi:10.1016/j.tourman.2024.105007 · Cited by 88
Israelsen, B. W., & Ahmed, N. R. (2019) · ACM Computing Surveys · doi:10.1145/3267338 · Cited by 74
“With the rapid advancement of intelligent technology to do tasks that were previously assumed to be too complicated for machines, there is now much discussion in public [24, 97, 110], business [6, 54, 101], and academic seings [12, 30] on how humans can trust said technology – although, the connection to trust is not…”
BackgroundZhaı, X., Shi, L., & Nehm, R. H. (2020) · Journal of Science Education and Technology · doi:10.1007/s10956-020-09875-z · Cited by 97
“The “black-box” problem of applied ML work (i.e., explaining the reasons behind successful or unsuccessful modeling; see Castelvecchi 2016) remains an intrinsic challenge.”
BackgroundOpportunities and Challenges: Classification of Skin Disease Based on Deep Learning
Zhang, B., Zhou, X. M., Luo, Y., Zhang, H., Yang, H., Ma, J., & Ma, L. (2021) · Chinese Journal of Mechanical Engineering · doi:10.1186/s10033-021-00629-5 · Cited by 95
“Moreover, it is possible that humans could not truly understand how a machine functions, even though it is actually inspired by humans [99, 100].”
BackgroundFairness Perceptions of Artificial Intelligence: A Review and Path Forward
Narayanan, D., Nagpal, M., McGuire, J., Schweitzer, S., & De Cremer, D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2210890 · Cited by 80
“In addition, the opaque nature of more complex AI models – such as those based on deep learning techniques – may also call into question the correctability of such decision processes (Castelvecchi, 2016).”
BackgroundA Survey of Autonomous Driving from a Deep Learning Perspective
Zhao, J., Wu, Y., Deng, R., Xu, S., Gao, J., & Burke, A. F. (2025) · ACM Computing Surveys · doi:10.1145/3729420 · Cited by 150
Velmovitsky, P. E., Bevilacqua, T., Alencar, P., Cowan, D. D., & Morita, P. P. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.561873 · Cited by 111
“In this technique, small computational units (the “neurons”) are arranged in layers that are connected (digital “synapses”) (91).”
BackgroundMethodIndividual mobility prediction review: Data, problem, method and application
Ma, Z., & Zhang, P. (2022) · Multimodal Transportation · doi:10.1016/j.multra.2022.100002 · Cited by 87
Human age reversal: Fact or fiction?
Johnson, A. A., English, B. W., Shokhirev, M. N., Sinclair, D. A., & Cuellar, T. L. (2022) · Aging Cell · doi:10.1111/acel.13664 · Cited by 89
“Since artificial intelligence is often conceptualized as a mysterious dark box (Castelvecchi, 2016) and lies at the heart of aging clocks, we will briefly introduce how machine learning models identify and utilize features for age prediction.”
BackgroundDeep learning for wildfire risk prediction: Integrating remote sensing and environmental data
Xu, Z., Li, J., Cheng, S. L., Xue, R., Zhao, Y., He, H., Guan, H., Sharma, A., Erxleben, M., Chang, R., et al. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.06.002 · Cited by 130
Large language models and multimodal foundation models for precision oncology
Truhn, D., Eckardt, J.-N., Ferber, D., & Kather, J. N. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00573-2 · Cited by 79
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘ black boxes ’ 27,28 .”
BackgroundLee, R. Y., Kross, E. K., Torrence, J., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Engelberg, R. A., & Curtis, J. R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1204 · Cited by 74
Recent Radiomics Advancements in Breast Cancer: Lessons and Pitfalls for the Next Future
Pesapane, F., Rotili, A., Agazzi, G. M., Botta, F., Raimondi, S., Penco, S., Dominelli, V., Cremonesi, M., Jereczek‐Fossa, B. A., Carrafiello, G., et al. (2021) · Current Oncology · doi:10.3390/curroncol28040217 · Cited by 83
“Data would need to be anonymized as the rules of patient privacy and the cybersecurity measures will be increasingly important in radiomic research [1,119,120].”
BackgroundProvenance documentation to enable explainable and trustworthy AI: A literature review
Kale, A., Nguyen, T., Harris, F. C., Li, C., Zhang, J., & Ma, X. (2023) · Data Intelligence · doi:10.1162/dint_a_00119 · Cited by 77
Emerging technologies for behavioral research in changing environments
Couzin, I. D., & Heins, C. (2022) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2022.11.008 · Cited by 95
The Impact of Artificial Intelligence on Microbial Diagnosis
Alsulimani, A., Akhter, N., Jameela, F., Ashgar, R. I., Jawed, A., Hassani, M. A., & Dar, S. A. (2024) · Microorganisms · doi:10.3390/microorganisms12061051 · Cited by 111
Huang, L., Liu, L., Jiang, L., & Zhang, T. (2018) · Remote Sensing · doi:10.3390/rs10122067 · Cited by 82
Tong, T., Gu, J., Xu, D., Song, L., Zhao, Q., Cheng, F., Yuan, Z., Tian, S., Yang, X., Tian, J., et al. (2022) · BMC Medicine · doi:10.1186/s12916-022-02258-8 · Cited by 77
“One major reason is that the interpretability of deep learning features is still in its infancy [49, 50], and the biological mechanism behind these radiomics features remain underexplored.”
BackgroundGenerating Counterfactual and Contrastive Explanations using SHAP
Rathi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09293 · Cited by 51
“AI often has a critique that when effective, it is a black box [Castelvecchi, 2016].”
BackgroundArtificial Intelligence in Emergency Medicine: Surmountable Barriers With Revolutionary Potential
Grant, K. L., McParland, A., Mehta, S., & Ackery, A. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2019.12.024 · Cited by 84
Wang, Y., Jiang, J., Li, S., Li, R., Xu, S., Wang, J., & Li, K. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3275792 · Cited by 101
“Restrictions apply. error [153], [154].”
Mesejo, P., Martos, R., Ibáñez, O. C., Novo, J., & Ortega, M. (2020) · Applied Sciences · doi:10.3390/app10144703 · Cited by 92
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.