Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Items 201–300 of 1,006
Artificial intelligence in nuclear industry: Chimera or solution?
Suman, S. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.124022 · Cited by 87
Toderaş, M. (2025) · Sustainability · doi:10.3390/su17178049 · Cited by 124
Prospects of deep learning for medical imaging
Kim, J., Hong, J., & Park, H. (2018) · Precision and Future Medicine · doi:10.23838/pfm.2018.00030 · Cited by 89
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96
“As the neural networks become more complex, the solutions they provide and how they come to those solutions becomes increasingly difficult to interpret [204].”
BackgroundVaghefi, E., Hill, S., Kersten, H. M., & Squirrell, D. (2020) · Journal of Ophthalmology · doi:10.1155/2020/7493419 · Cited by 87
“+e black box nature of the neural networks makes any interpretation of the results problematic [43].”
BackgroundImpacts of Attitudes Toward Government and Corporations on Public Trust in Artificial Intelligence
Chen, Y.-N. K., & Wen, C.-H. R. (2021) · Communication Studies · doi:10.1080/10510974.2020.1807380 · Cited by 73
“The other reason is that there are still many inscrutable puzzles beyond humans in the field of AI such as block box problems and ethical issues (Castelvecchi, 2016), and scientists are reluctant to address them until there is a concrete policy in place (Deng, 2015; Gulson & Webb, 2017; Neri &…”
BackgroundZając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 74
“Unlike rule-based algorithms, whose decision-making processes can be inspected, ML models are often criticised for the “black-box” problem [27], i.”
BackgroundStandards, frameworks, and legislation for artificial intelligence (AI) transparency
Lund, B. D., Orhan, Z., Mannuru, N. R., Bevara, R. V. K., Porter, B., Vinaih, M. K., & Bhaskara, P. (2025) · AI and Ethics · doi:10.1007/s43681-025-00661-4 · Cited by 137
“AI’s “black box” dilemma, where no transparency exists about how these tools operate, diminishes public trust and developer accountability [ 12].”
Interpretable deep learning for guided microstructure-property explorations in photovoltaics
Pokuri, B. S. S., Ghosal, S., Kokate, A., Sarkar, S., & Ganapathysubramanian, B. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0231-y · Cited by 73
“model that is not interpretable raises a critical question—why should a domain expert believe in the prediction of a black-box model? This lack of “interpretability or explainability” is endemic to most black-box models and presents a major bottleneck to the widespread acceptance of ML models.(23) Recently, there have…”
MethodAI-enabled business models for competitive advantage
Climent, R. C., Haftor, D., & Staniewski, M. W. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100532 · Cited by 133
“Instead, it is conditioned by patterns identi fi ed in the data (Castelvecchi, 2016; Kemp, 2023; Murray, Rhymer & Sirmon 2021).”
BackgroundQuinn, J. D., Reed, P. M., Giuliani, M., & Castelletti, A. F. (2019) · Water Resources Research · doi:10.1029/2018wr024177 · Cited by 72
“, 2018), a common critique is that the resulting control policy functions are “opaque,” or difficult to interpret (Alexander & Mozer, 1999; Castelvecchi, 2016).”
BackgroundMethodRole of artificial intelligence in MS clinical practice
Bonacchi, R., Filippi, M., & Rocca, M. A. (2022) · NeuroImage Clinical · doi:10.1016/j.nicl.2022.103065 · Cited by 86
“This is an active area of research in computer science (Castelvecchi, 2016), which could lead to major breakthroughs in the medical field.”
Li, G., Wang, L., Shen, L., Chen, L., Cheng, H.-D., Xu, C., & Li, F. (2023) · Energy and Buildings · doi:10.1016/j.enbuild.2023.112949 · Cited by 83
Towards big industrial data mining through explainable automated machine learning
Garouani, M., Ahmad, A., Bouneffa, M., Hamlich, M., Bourguin, G., & Lewandowski, A. (2022) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-022-08761-9 · Cited by 92
“the complexity of the AI algorithms and systems grows, they are viewed as “black-boxes” [27, 32].”
BackgroundKoçak, B. (2022) · Diagnostic and Interventional Radiology · doi:10.5152/dir.2022.211297 · Cited by 85
The Enlightening Role of Explainable Artificial Intelligence in Chronic Wound Classification
Sarp, S., Kuzlu, M., Wilson, E., Cali, Ü., & Güler, Ö. (2021) · Electronics · doi:10.3390/electronics10121406 · Cited by 94
“When an AI model learns and gives an output, it processes the data and deciphers the processed information immediately instead of storing the learned data as a clear digital memory [7].”
MethodPesapane, F., Bracchi, D., Mulligan, J. F., Linnikov, A. S., Maslennikov, O. V., Lanzavecchia, M. B., Tantrige, P. M., Stasolla, A., Biondetti, P., Giuggioli, P. F., et al. (2021) · Radiation · doi:10.3390/radiation1040022 · Cited by 93
“Considering that the amount of data and the number of AI applications can only grow, regulatory actions regarding cybersecurity will face continuous challenges [74].”
BackgroundNeto, S. R. D. S., de Oliveira, T. T., Teixeira, I. V., de Oliveira, S. B. A., Sampaio, V. D. S., Lynn, T., & Endo, P. T. (2022) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0010061 · Cited by 83
Chen, Y., Gu, X., Liu, Z., & Jia, L. (2022) · Remote Sensing · doi:10.3390/rs14081877 · Cited by 92
Recent Advances in Deep-Learning-Based SAR Image Target Detection and Recognition
Lang, P., Fu, X., Dong, J., Yang, H., Yin, J., Yang, J., & Martorella, M. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3543531 · Cited by 91
“1) Physically Explainable DL for SAR-TDR: The black box property of the DNN model [264] is still a severe challenge in practical DL-based SAR-TDR application domains.”
Using Machine Learning to Assess Physician Competence: A Systematic Review
Dias, R. D., Gupta, A., & Yule, S. J. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002414 · Cited by 70
SAR Oil Spill Detection System through Random Forest Classifiers
Conceição, M. R. A., de Mendonça, L. F. F., Lentini, C. A. D., Lima, A. T. D. C., Lopes, J. M., de Vasconcelos, R. N., Gouveia, M. B., & Porsani, M. J. (2021) · Remote Sensing · doi:10.3390/rs13112044 · Cited by 83
“As Michael Tyka says in [44], “the problem is that the knowledge gets baked into the network, rather than into us”.”
BackgroundA locally-constrained YOLO framework for detecting small and densely-distributed building footprints
Xie, Y., Cai, J., Bhojwani, R., Shekhar, S., & Knight, J. (2019) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2019.1624761 · Cited by 76
“This ” black-box ” nature of deep learning still needs to be addressed or reduced (Castelvecchi 2016).”
BackgroundExplaining sentiment analysis results on social media texts through visualization
Jain, R., Kumar, A., Nayyar, A., Dewan, K., Garg, R., Raman, S., & Ganguly, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14432-y · Cited by 68
“They are closely related and essential to “open the black-box” of machine learning models [13, 47].”
BackgroundSchultz, M. D., & Seele, P. (2022) · AI and Ethics · doi:10.1007/s43681-022-00150-y · Cited by 96
“A core challenge of AI ethics is the opacity and obfuscation of algorithms and third-party data collection [22, 40].”
BackgroundSecurity Considerations in AI-Robotics: A Survey of Current Methods, Challenges, and Opportunities
Neupane, S., Mitra, S., Fernandez, I. A., Saha, S., Mittal, S., Chen, J., Pillai, N., & Rahimi, S. (2024) · IEEE Access · doi:10.1109/access.2024.3363657 · Cited by 114
“AI components that involve deep learning (i.e. neural networks) can be especially vulnerable due to their “black box” nature, which means that even when the security is compromised, it may be difficult to pinpoint exactly where the problem occurred and how to address it [156].”
BackgroundTrustworthy remote sensing interpretation: Concepts, technologies, and applications
Wang, S., Han, W., Zhang, X., Zhang, X., Wang, L., & Li, J. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.02.003 · Cited by 84
An artificial neural network based approach to investigate travellers’ decision rules
van Cranenburgh, S., & Alwosheel, A. (2018) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2018.11.014 · Cited by 68
“Future research may be directed to illuminate the black boxes of ANNs (Castelvecchi, 2016).”
BackgroundEngineering Equity: How AI Can Help Reduce the Harm of Implicit Bias
Lin, Y.-T., Hung, T.-W., & Huang, L. T.-L. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00406-7 · Cited by 97
“Additionally, the manner by which ML generates results is often opaque and inexplicable (Castelvecchi 2016; Wachter et al. 2017).”
BackgroundInfluentialProfessional learning networks: a conceptual model and research opportunities
Poortman, C. L., Brown, C., & Schildkamp, K. (2021) · Educational Research · doi:10.1080/00131881.2021.1985398 · Cited by 105
“Systems exist that can discover patterns in large amounts of data, and these systems can replicate human reasoning, such as arguing, generalising, interpreting and learning from the past (Castelvecchi 2016).”
BackgroundBaquer, G., Sementé, L., Mahamdi, T., Correig, X., Ràfols, P., & García‐Altares, M. (2022) · Mass Spectrometry Reviews · doi:10.1002/mas.21794 · Cited by 71
Padhi, A., Agarwal, A., Saxena, S. K., & Katoch, C. D. S. (2023) · VirusDisease · doi:10.1007/s13337-023-00841-y · Cited by 75
“AI models, especially deep learning ones, are often termed “black boxes,” making it difficult to understand how they make decisions [4].”
BackgroundSystems metabolomics: from metabolomic snapshots to design principles
Damiani, C., Gaglio, D., Sacco, E., Alberghina, L., & Vanoni, M. (2020) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2020.02.013 · Cited by 75
“Moreover, machine learning algorithms are characterized by the well-known black box problem, which makes it difficult to know how and why they work [39].”
BackgroundGrzybowski, A., & Brona, P. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10112352 · Cited by 66
Odenkirk, M. T., Reif, D. M., & Baker, E. S. (2021) · Analytical Chemistry · doi:10.1021/acs.analchem.0c04850 · Cited by 74
Dumić-Čule, I., Orešković, T., Brkljačić, B., Kujundžić Tiljak, M., & Orešković, S. (2020) · Croatian Medical Journal · doi:10.3325/cmj.2020.61.457 · Cited by 61
Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry
Cong, S., Wang, H., Zhou, Y., Wang, Z., Yao, X., & Yang, C. (2024) · Brain‐X · doi:10.1002/brx2.57 · Cited by 81
Alsabah, M. Q., Naser, M. A., Albahri, A. S., Albahri, A. S., Alamoodi, A. H., Abdulhussain, S. H., & Alzubaidi, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11342-3 · Cited by 109
Burlacu, A., Iftene, A., Jugrin, D., Popa, I. V., Lupu, P. M., Vlad, C.-E., & Covic, A. C. (2020) · BioMed Research International · doi:10.1155/2020/9867872 · Cited by 74
“Since ML algorithms have the strengths to learn and improve from experience without being explicitly programmed for a speci fi c task, AI is often perceived as a “ black box ” that hides the precise way in which it concludes/results [72].”
Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
Eitel, F., Schulz, M.-A., Seiler, M., Walter, H., & Ritter, K. (2021) · Experimental Neurology · doi:10.1016/j.expneurol.2021.113608 · Cited by 79
“Despite their recent success, ANNs, and in particular, deep neural networks, are often criticized for being black-boxes [93, 94].”
BackgroundCiallella, H. L., Russo, D. P., Aleksunes, L. M., Grimm, F. A., & Zhu, H. (2021) · Environmental Science & Technology · doi:10.1021/acs.est.1c02656 · Cited by 63
OD-XAI: Explainable AI-Based Semantic Object Detection for Autonomous Vehicles
Mankodiya, H., Jadav, D., Gupta, R., Tanwar, S., Hong, W., & Sharma, R. (2022) · Applied Sciences · doi:10.3390/app12115310 · Cited by 86
Explainable Artificial Intelligence for Cybersecurity
Sharma, D. K., Mishra, J., Singh, A., Govil, R., Srivastava, G., & Lin, J. C. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108356 · Cited by 88
Wang, G., Guo, Y., Zhang, W., Xie, S., & Chen, Q. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101803 · Cited by 78
Tracing the legitimacy of Artificial Intelligence: A longitudinal analysis of media discourse
Korneeva, E., Salge, T. O., Teubner, T., & Antons, D. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122467 · Cited by 93
Understanding How People Reason about Aesthetic Evaluations of Artificial Intelligence
Oh, C., Kim, S., Choi, J., Eun, J., Kim, S., Kim, J., Lee, J., & Suh, B. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395430 · Cited by 75
“Despite the remarkable advances of AI algorithms with the development of deep learning (DL), it has been pointed out that it is relatively difficult to understand how the internal principles and mechanisms of the algorithms work [3, 8, 32].”
BackgroundZou, Y., Shi, Y., Sun, F., Liu, J., Guo, Y. L., Zhang, H., Lu, X., Gong, Y., & Xia, S. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107038 · Cited by 70
Conceptualizing generative AI as style engines: Application archetypes and implications
Riemer, K., & Peter, S. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102824 · Cited by 63
Anguita-Ruiz, A., Segura-Delgado, A., Alcalá, R., Aguilera, C. M., & Alcalá-Fdez, J. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007792 · Cited by 62
Lin, C., Chau, T., Lin, C.-S., Shang, H.-S., Fang, W.-H., Lee, D.-J., Lee, C.-C., Tsai, S.-H., Wang, C.-H., & Lin, S.-H. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00550-0 · Cited by 62
“K + being more transparent 35 .”
Yildiz, V., Tian, P., Yildiz, I., Brown, J. M., Kalpathy-Cramer, J., Dy, J. G., Ioannidis, S., Erdoğmuş, D., Ostmo, S., Kim, S. J., et al. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.10 · Cited by 59
The future of phenomics in dairy cattle breeding
Cole, J. B., Eaglen, S., Maltecca, C., Mulder, H. A., & Pryce, J. E. (2020) · Animal Frontiers · doi:10.1093/af/vfaa007 · Cited by 78
“However, these new approaches have their own challenges, ranging from bias (Castelvecchi, 2016) to interpretability (Gilpin et al., 2018), and there is a temptation to oversell outcomes.”
BackgroundEnhancing Kidney Transplant Care through the Integration of Chatbot
Valencia, O. A. G., Thongprayoon, C., Jadlowiec, C. C., Mao, S. A., Miao, J., & Cheungpasitporn, W. (2023) · Healthcare · doi:10.3390/healthcare11182518 · Cited by 74
“Striving for interpretability and explainability is crucial to fostering trust, accountability, and acceptance of the chatbot in kidney transplant care.”
Herrera, F. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103133 · Cited by 79
Raket, L. L., Jaskolowski, J., Kinon, B. J., Brasen, J. C., Jönsson, L., Wehnert, A., & Fusar-Poli, P. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30024-8 · Cited by 60
Luppi, A. I., Cain, J. A., Spindler, L. R. B., Górska, U., Toker, D., Hudson, A. E., Brown, E. N., Diringer, M. N., Stevens, R. D., Massimini, M., et al. (2021) · Neurocritical Care · doi:10.1007/s12028-021-01281-6 · Cited by 70
“, consciousness) and what is being measured; a similar problem is the so-called black box in medical artificial intelligence [135], whereby classification (diagnosis) and prediction (prognosis) can be successful but for opaque reasons [135].”
BackgroundExplainability and white box in drug discovery
Kırboğa, K. K., Abbasi, S. W., & Küçüksille, E. U. (2023) · Chemical Biology & Drug Design · doi:10.1111/cbdd.14262 · Cited by 68
“This free service is offered worldwide and is utilized daily by over 200 million individuals (Castelvecchi, 2016).”
BackgroundMcNeese, N. J., Flathmann, C., O’Neill, T., & Salas, E. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107874 · Cited by 74
Machine learning for radiation outcome modeling and prediction
Luo, Y., Chen, S., & Valdés, G. (2020) · Medical Physics · doi:10.1002/mp.13570 · Cited by 63
Bourdenx, M., Nioche, A., Doveró, S., Arotçarena, M.-L., Camus, S. M. J., Porras, G., Thiolat, M.-L., Rougier, N. P., Prigent, A., Aubert, P., et al. (2020) · Science Advances · doi:10.1126/sciadv.aaz9165 · Cited by 70
“A major issue in the use of machine learning in experimental biology in the “black box” is the fact that it is usually impossible to “understand” how an algo-rithm predicts an output ( 44 ).”
Impact of Artificial Intelligence on Medical Education in Ophthalmology
Valikodath, N. G., Cole, E., Ting, D. S. W., Campbell, J. P., Pasquale, L. R., Chiang, M. F., Chan, R. V. P., & on behalf of the American Academy of Ophthalmology Task Force on Artificial Intelligence (2021) · Translational Vision Science & Technology · doi:10.1167/tvst.10.7.14 · Cited by 59
Deep Learning for Genomics: From Early Neural Nets to Modern Large Language Models
Yue, T., Wang, Y., Zhang, L., Gu, C., Xue, H., Wang, W., Lyu, Q., & Dun, Y. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms242115858 · Cited by 197
“Connections among network units are so convo-luted that the information is widespread throughout the network and thus perplexing to be captured [110].”
BackgroundAn explanation framework and method for AI-based text emotion analysis and visualisation
Li, Y., Chan, J., Peko, G. M., & Sundaram, D. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.114121 · Cited by 78
Artificial intelligence in COPD CT images: identification, staging, and quantitation
Wu, Y., Xia, S., Liang, Z., Chen, R., & Qi, S. (2024) · Respiratory Research · doi:10.1186/s12931-024-02913-z · Cited by 71
“Many AI models, particularly deep learning algorithms, operate as “black boxes”, making it difficult for clinicians to understand how the model arrived at a particular decision [164].”
BackgroundGrote, G., Parker, S. K., & Crowston, K. (2024) · Academy of Management Review · doi:10.5465/amr.2023.0117 · Cited by 116
Machine Learning for Accurate Intraoperative Pediatric Middle Ear Effusion Diagnosis
Crowson, M. G., Hartnick, C. J., Diercks, G. R., Gallagher, T. Q., Fracchia, M. S., Setlur, J., & Cohen, M. S. (2021) · PEDIATRICS · doi:10.1542/peds.2020-034546 · Cited by 69
Controlling Safety of Artificial Intelligence-Based Systems in Healthcare
Davahli, M. R., Karwowski, W., Fiok, K., Wan, T. T. H., & Parsaei, H. R. (2021) · Symmetry · doi:10.3390/sym13010102 · Cited by 72
Kim, K., Yang, H., Yi, J., Son, H. E., Ryu, J. Y., Kim, Y. C., Jeong, J. C., Chin, H. J., Na, K. Y., Chae, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/24120 · Cited by 64
“In this respect, neural network models have been criticized as being black-box models [26].”
MethodElul, Y., Rosenberg, A. A., Schuster, A., Bronstein, A. M., & Yaniv, Y. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2020620118 · Cited by 65
“While common opinion holds that DL algorithms are uninterpretable “ black-boxes ” (11, 17, 18), this depends on what is required to be explained, with no lack of evidence that deep models are, in some respects, more interpretable than traditional models (16, 19).”
BackgroundThe Impact of Artificial Intelligence on Sustainable Development in Electronic Markets
Thamik, H., & Wu, J. (2022) · Sustainability · doi:10.3390/su14063568 · Cited by 79
Artificial Intelligence in Endodontic Education
Aminoshariae, A., Nosrat, A., Nagendrababu, V., Dianat, O., Mohammad‐Rahimi, H., O'Keefe, A. W., & Setzer, F. C. (2024) · Journal of Endodontics · doi:10.1016/j.joen.2024.02.011 · Cited by 96
Lan, G., Feng, X., Du, S., Song, F., & Xiao, Q. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13427-6 · Cited by 92
“However, despite their potential benefits, GenAI tools also present new chal - lenges that necessitate a robust understanding of both the technology and its ethical implications (Castelvecchi, 2016).”
Machine learning for the meta-analyses of microbial pathogens’ volatile signatures
Palma, S., Traguedo, A. P., Porteira, A. R., Frias, M. J., Gamboa, H. F. S., & Roque, A. C. A. (2018) · Scientific Reports · doi:10.1038/s41598-018-21544-1 · Cited by 65
Knowledge in the grey zone: AI and cybersecurity
Stevens, T. (2020) · Digital War · doi:10.1057/s42984-020-00007-w · Cited by 62
“Hidden within the archetypal ‘black box’ of technology, defined in terms of inputs and outputs only (Von Hilgers 2011), the computation of AI technologies is frequently unknowable and therefore radically opaque (Castelvecchi 2016).”
BackgroundZhang, G., Shi, Y., Yin, P., Liu, F., Fang, Y., Li, X., Zhang, Q., & Zhang, Z. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.944569 · Cited by 62
“Due to the complex nonlinear relationship of some ML algorithms, the model results are dif fi cult to interpret, resulting in a “ black-box ” problem (10), which limits the clinical application of predictive models.”
MethodA scoping review of neurodegenerative manifestations in explainable digital phenotyping
Alfalahi, H., Dias, S. B., Khandoker, A. H., Chaudhuri, К. R., & Hadjileontiadis, L. J. (2023) · npj Parkinson s Disease · doi:10.1038/s41531-023-00494-0 · Cited by 67
“By mimicking how the brain learns, arti fi cial neural networks are set at the center of ef fi cient “ big-data ” analysis by tweaking the strength of connections between multi-layer neurons during training, ultimately leading to robust classi fi cation and/or regression performance 216 .”
MethodAdibnia, E., Mansouri-Birjandi, M. A., Ghadrdan, M., & Jafari, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-56522-3 · Cited by 63
Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62
“ML algorithms are often associated with the so-called “black box” effect [11,12].”
Shu, S., Ren, J., & Song, J. (2021) · Circulation Journal · doi:10.1253/circj.cj-20-1121 · Cited by 66
Harm to Nonhuman Animals from AI: a Systematic Account and Framework
Coghlan, S., & Parker, C. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00627-6 · Cited by 75
“Consequently, harmful predictions made by this ‘black-box’ AI (Castelvecchi, 2016) can sometimes be difficult or impossible to detect and prevent.”
BackgroundRecent Advances in Explainable Artificial Intelligence for Magnetic Resonance Imaging
Qian, J., Li, H., Wang, J., & He, L. (2023) · Diagnostics · doi:10.3390/diagnostics13091571 · Cited by 64
Leveraging machine vision in cell-based diagnostics to do more with less
Doan, M., & Carpenter, A. E. (2019) · Nature Materials · doi:10.1038/s41563-019-0339-y · Cited by 55
Producing Radiologist-Quality Reports for Interpretable Deep Learning
Gale, W., Oakden-Rayner, L., Carneiro, G., Palmer, L. J., & Bradley, A. P. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759236 · Cited by 44
“In response to these concerns, researchers have sought to “open the black box of AI” [5] by attempting to explain the decisions of systems in a variety of ways, typically falling into the following categories: 1) identifying regions of the image that contribute to the decision (saliency…”
BackgroundArtificial Intelligence in Reproductive Medicine – An Ethical Perspective
Rolfes, V., Bittner, U., Gerhards, H., Krüssel, J.-S., Fehm, T., Ranisch, R., & Fangerau, H. (2023) · Geburtshilfe und Frauenheilkunde · doi:10.1055/a-1866-2792 · Cited by 53
Gao, Y., Miyata, S., & Akashi, Y. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121591 · Cited by 63
Mastropietro, A., Feldmann, C., & Bajorath, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-46930-2 · Cited by 60
An Ethical Perspective on the Democratization of Mental Health With Generative AI
Elyoseph, Z., Gur, T., Haber, Y., Simon, T., Angert, T., Navon, Y., Tal, A., & Asman, O. (2024) · JMIR Mental Health · doi:10.2196/58011 · Cited by 80
“Indeed, the algorithms driving this AI application function are much like a “black box” shrouded in mystery with no clarity as to how determinations and analyses are reached [36].”
BackgroundInterpretable and explainable predictive machine learning models for data-driven protein engineering
Medina-Ortiz, D., Khalifeh, A., Anvari-Kazemabad, H., & Davari, M. D. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108495 · Cited by 63
Integrated explainable deep learning prediction of harmful algal blooms
Lee, D.-H., Kim, M., Lee, B., Chae, S., Kwon, S., & Kang, S. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122046 · Cited by 60
Explainable deep learning for automatic rock classification
Zheng, D., Zhong, H., Camps‐Valls, G., Cao, Z., Ma, X., Mills, B. J. W., Hu, X., Hou, M., & Ma, C. (2023) · Computers & Geosciences · doi:10.1016/j.cageo.2023.105511 · Cited by 55
Cancer subtype classification and modeling by pathway attention and propagation
Lee, S., Lim, S., Lee, T., Sung, I., & Kim, S. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa203 · Cited by 64
A systematic mapping on automatic classification of fake news in social media
de Souza, J. V., Gomes, J., Filho, F. M. D. S., de Oliveira, A. M., & de Souza, J. F. (2020) · Social Network Analysis and Mining · doi:10.1007/s13278-020-00659-2 · Cited by 62
“Although the number of neural-based approaches has increased after 2016, some authors state that it is crucial to understand the results of the model used, and it is not always possible in neural models (the called black-box model Castelvecchi 2016; Knight 2017).”
BackgroundMass-scale emotionality reveals human behaviour and marketplace success
Rocklage, M. D., Rucker, D. D., & Nordgren, L. F. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01098-5 · Cited by 52
Nazari, M., Kluge, A., Apostolova, I. I., Klutmann, S., Kimiaei, S., Schroeder, M., & Buchert, R. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05569-9 · Cited by 56
Valeriani, D., & Simonyan, K. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2009165117 · Cited by 59
“Finally, the visualization of the identified microstructural neural network biomarker and the interpretability of DystoniaNet’s internal model for automated diagnostic decision-making offers a transparent and fair machinelearning platform and makes its translation to healthcare more compliant (48, 49).”
Geoscience-aware deep learning: A new paradigm for remote sensing
Ge, Y., Zhang, X., Atkinson, P. M., Stein, A., & Li, L. (2022) · Science of Remote Sensing · doi:10.1016/j.srs.2022.100047 · Cited by 64
Towards Explainability for AI Fairness
Zhou, J., Chen, F., & Holzinger, A. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_18 · Cited by 43
Liu, M., Zhang, H., Liu, Z., & Zhao, N. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3179491 · Cited by 54
“Despite the unique advantages of DL in solving radio communication problems, the black-box and unexplained properties of deep neural networks (DNNs) [11] may lead to many security risks.”
BackgroundEmerging role of artificial intelligence in cardiac electrophysiology
Kabra, R., Israni, S., Vijay, B., Baru, C. K., Mendu, R., Fellman, M., Sridhar, A. R. M., Mason, P. K., Cheung, J. W., Biase, L. D., et al. (2022) · Cardiovascular Digital Health Journal · doi:10.1016/j.cvdhj.2022.09.001 · Cited by 56
Neural network in food analytics
Ma, P., Zhang, Z., Jia, X., Peng, X., Zhang, Z., Tarwa, K., Wei, C., Liu, F., & Wang, Q. (2022) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2022.2139217 · Cited by 59
“Besides, there is no guarantee of the model output will be consistently correct when presented with the same input due to stochastic parameter generation (Castelvecchi 2016).”
Background
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.