Can we open the black box of AI?
Open accessThe DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“most contemporary AI models operate as black boxes, making their internal processes difficult to comprehend (Castelvecchi 2016).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Castelvecchi 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The latter space comprises hundreds of layers and millions of parameters, which makes DNNs be considered as complex black-box models [4].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Lastly, a prominent concern of AI models based on neural network architecture is their often criticized lack of interpretability which earned them the term ‘black boxes’27,28.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Most of these discussions point to the fact that algorithmic decision-making is often difficult to understand for - and explain to - human agents, due to the different, and sometimes inscrutable for persons, ways of AI operations: the so-called 'black box' problem (Castelvecchi, 2016).”
From Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021) · cites this paper as Castelvecchi, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 2,890
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755
“The much-debated (Press, 2016) Only Humans Need Apply is the latest book on AI by Babson Professor Thomas H. Davenport and Harvard Business Review Contributing Editor Julia Kirby (Davenport & Kirby, 2016).”
BackgroundAn overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058
“This is the “black box” problem of deep neural networks [344].”
BackgroundAdversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733
Artificial Intelligence and Deep Learning in Ophthalmology
Wang, Z., Keane, P. A., Chiang, M., Cheung, C. Y., Wong, T. Y., & Ting, D. S. W. (2022) · Artificial Intelligence in Medicine · doi:10.1007/978-3-030-64573-1_200 · Cited by 1,524
Artificial intelligence and deep learning in ophthalmology
Ting, D. S. W., Pasquale, L. R., Peng, L., Campbell, J. P., Lee, A. Y., Raman, R., Tan, G. S. W., Schmetterer, L., Keane, P. A., & Wong, T. Y. (2019) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2018-313173 · Cited by 1,554
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
“This process is coupled with the deep dream technique to map the network’s response to any given image [14].”
BackgroundArtificial Intelligence and the Public Sector—Applications and Challenges
Wirtz, B. W., Weyerer, J. C., & Geyer, C. (2019) · International Journal of Public Administration · doi:10.1080/01900692.2018.1498103 · Cited by 1,341
“In this connection, Castelvecchi (2016, p. 21) highlights the opaqueness of AI systems and notes that “ [i]nstead of storing what they have learned in a neat block of digital memory, they diffuse the information in a way that is exceedingly difficult to decipher, ” which in turn makes them also…”
BackgroundSpeech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2018) · NORA - Norwegian Open Research Archives · Cited by 1,012
Celik, I. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107468 · Cited by 1,012
Multi-faceted particle pumps drive carbon sequestration in the ocean
Boyd, P. W., Claustre, H., Levy, M., Siegel, D. A., & Weber, T. (2019) · Nature · doi:10.1038/s41586-019-1098-2 · Cited by 894
Schneider, G. (2018) · Nature Reviews Drug Discovery · doi:10.1038/nrd.2017.232 · Cited by 910
Managing artificial intelligence
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16274 · Cited by 898
FinBERT: A Large Language Model for Extracting Information from Financial Text*
Huang, A. H., Wang, H., & Yang, Y. (2023) · Contemporary Accounting Research · doi:10.1111/1911-3846.12832 · Cited by 778
The impact of artificial intelligence on learner–instructor interaction in online learning
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-021-00292-9 · Cited by 788
“For instance, when communicating with an AI Teaching Assistant, the black-box nature of the AI system leaves no choices for students to check whether the answers from AI are right or wrong (Castelvecchi, 2016).”
BackgroundTransparency and the black box problem: why we do not trust AI
von Eschenbach, W. J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00477-0 · Cited by 786
“Certainly, comparisons between DL’s neural networks and human brains can be made, and, in many ways, human cognition can be viewed as a black box as well (Burrell, 2016; Castelvecchi, 2016).”
BackgroundAI revolutionizing industries worldwide: A comprehensive overview of its diverse applications
Rashid, A. B., & Kausik, M. A. K. (2024) · Hybrid Advances · doi:10.1016/j.hybadv.2024.100277 · Cited by 764
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
“AI systems are becoming more complicated, making them challenging to understand [37].”
BackgroundIntroducing artificial intelligence training in medical education
Paranjape, K., Schinkel, M., Nannan Panday, R., Car, J., & Nanayakkara, P. (2019) · JMIR Medical Education · doi:10.2196/16048 · Cited by 634
“Along with changing the conventional ways physician work, the black box problem [29] and liability issues [30] are some of the most anticipated challenges.”
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606
“However, because of the black box nature of ML models [6,7], the deployment of ML algorithms, especially in high stake domains, such as medical diagnosis, criminal justice, financial decision-making, and other regulated safety critical domains, requires verification and testing for plausibility by…”
BackgroundGenerative artificial intelligence
Banh, L., & Strobel, G. (2023) · Electronic Markets · doi:10.1007/s12525-023-00680-1 · Cited by 579
“…is considered an umbrella term, spanning over different computational algorithms capable of performing tasks that typically require human intelligence, such as understanding natural language, recognizing patterns, making decisions, and learning from experience (Castelvecchi, 2016; Winston, 1993).”
BackgroundSwanson, K., Wu, E., Zhang, A., Alizadeh, A. A., & Zou, J. (2023) · Cell · doi:10.1016/j.cell.2023.01.035 · Cited by 597
“, from1 to10) shouldbecarefully designed and calibrated to user expectations.(128)”
BackgroundSurvey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
“Furthermore, the potential risks in big data algorithms that cannot be ignored allow for increased bias and replication or exacerbate human errors [23].”
BackgroundAn explainable deep machine vision framework for plant stress phenotyping
Ghosal, S., Blystone, D., Singh, A. K., Ganapathysubramanian, B., Singh, A., & Sarkar, S. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1716999115 · Cited by 553
Grassmann, F., Mengelkamp, J., Brandl, C., Harsch, S., Zimmermann, M. E., Linkohr, B., Peters, A., Heid, I. M., Palm, C., & Weber, B. H. F. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.02.037 · Cited by 549
Deep learning for site safety: Real-time detection of personal protective equipment
Nath, N. D., Behzadan, A. H., & Paal, S. G. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103085 · Cited by 540
Chalmers, D., MacKenzie, N. G., & Carter, S. (2021) · Entrepreneurship Theory and Practice · doi:10.1177/1042258720934581 · Cited by 535
“A further danger for rapidly scaling companies can be found in the lack of transparency over how some deep learning systems arrive at their outputs (Castelvecchi, 2016).”
BackgroundAl’Aref, S. J., Anchouche, K., Singh, G., Slomka, P. J., Kolli, K. K., Kumar, A., Pandey, M., Maliakal, G., van Rosendael, A. R., Beecy, A. N., et al. (2019) · European Heart Journal · doi:10.1093/eurheartj/ehy404 · Cited by 534
“90 While a machine may be capable of accurately predicting an outcome of interest, the absence of a familiar logic underlying its output may give pause to the clinician charged with interpreting it.”
BackgroundAlwosheel, A., van Cranenburgh, S., & Chorus, C. G. (2018) · Journal of Choice Modelling · doi:10.1016/j.jocm.2018.07.002 · Cited by 524
“…and testing of rules-of-thumb for sample sizes in the context of ANN-based discrete choice analysis, we would like to note the following: ever since the introduction of ANNs, but especially in recent years (e.g., Castelvecchi, 2016), there has been debate about the ‘ black-box ’ -nature of ANNs.”
MethodAn Overview of Artificial Intelligence Ethics
Huang, C., Zhang, Z., Mao, B., & Yao, X. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3194503 · Cited by 482
“This mainly leads to the transparency issue [20].”
BackgroundPesapane, F., Volonté, C., Codari, M., & Sardanelli, F. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0645-y · Cited by 479
Schiff, D. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00270-2 · Cited by 489
Natural language processing applied to mental illness detection: a narrative review
Zhang, T., Schoene, A. M., Ji, S., & Ananiadou, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00589-7 · Cited by 476
“Nevertheless, they are still treated as black boxes 201 and fail to explain the predictions.”
BackgroundQiu, S., Joshi, P. S., Miller, M. I., Xue, C., Zhou, X., Karjadi, C., Chang, G. H., Joshi, A. S., Dwyer, B., Zhu, S., et al. (2020) · Brain · doi:10.1093/brain/awaa137 · Cited by 472
Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them
Santoni de Sio, F., & Mecacci, G. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00450-x · Cited by 490
“The way knowledge is represented in the machine and the exact way the machine distinguishes between a positive and negative diagnosis are not only inaccessible to the doctor who uses the sys-tem but also, to an important extent, to those who designed it (Castelvecchi, 2016).”
BackgroundLee, H., Yune, S., Mansouri, M., Kim, M., Tajmir, S. H., Guerrier, C. E., Ebert, S. A., Pomerantz, S. R., Romero, J. M., Kamalian, S., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0324-9 · Cited by 441
The Challenge of Machine Learning in Space Weather: Nowcasting and Forecasting
Camporeale, E. (2019) · Space Weather · doi:10.1029/2018sw002061 · Cited by 403
“However, as pointed out from Pierre Baldi, we trust our brain all the time, yet we have very little understanding of how it works (Castelvecchi, 2016).”
BackgroundO’Dwyer, E., Pan, I., Acha, S., & Shah, N. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.01.024 · Cited by 410
Machine learning-based approach: global trends, research directions, and regulatory standpoints
Pugliese, R., Regondi, S., & Marini, R. (2021) · Data Science and Management · doi:10.1016/j.dsm.2021.12.002 · Cited by 373
Competitive Advantage Through Artificial Intelligence: Toward a Theory of Situated AI
Kemp, A. (2024) · Academy of Management Review · doi:10.5465/amr.2020.0205 · Cited by 362
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
Speith, T. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534639 · Cited by 337
“to comprehend the inner workings of learned models [18, 20].”
BackgroundYang, C., Chen, M., & Yuan, Q. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106153 · Cited by 343
Tripathy, K. P., & Mishra, A. K. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.130458 · Cited by 336
Freeman, K., Geppert, J., Stinton, C., Todkill, D., Johnson, S., Clarke, A., & Taylor-Phillips, S. (2021) · BMJ · doi:10.1136/bmj.n1872 · Cited by 338
Understanding, explaining, and utilizing medical artificial intelligence
Cadario, R., Longoni, C., & Morewedge, C. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01146-0 · Cited by 342
Mahbooba, B., Timilsina, M., Sahal, R., & Serrano, M. (2021) · Complexity · doi:10.1155/2021/6634811 · Cited by 327
“'e combination of the parametric learning space comprises hundreds of layers and millions of parameters, which makes ML and DL models be considered as complex black-box models [7].”
BackgroundOut of the laboratory and into the classroom: the future of artificial intelligence in education
Schiff, D. (2021) · AI & SOCIETY · doi:10.1007/s00146-020-01033-8 · Cited by 334
“For example, do we expect or want students to be consciously aware of, or be able to decide whether a nudge occurs? Do we want nudges that harm, even in service of calculated long-term benefit, such as an AI system shaming a student for laziness or mistakes?21 More generally, is it appropriate to treat student emotion…”
BackgroundHan, W., Zhang, X., Wang, Y., Wang, L., Huang, X., Li, J., Wang, S., Chen, W., Li, X., Feng, R., et al. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.05.032 · Cited by 303
Liu, H., Li, L., Wormstone, I. M., Qiao, C., Zhang, C., Liu, P., Li, S., Wang, H., Mou, D., Pang, R., et al. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.3501 · Cited by 307
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
“The very power of such approaches (e.g., deep learning neural networks) lies in their ability to form thousands of extremely nuanced and complex rules resulting from millions or even billions of iterations over training data (Castelvecchi, 2016; Holzinger, 2016).”
Accelerating Climate Resilient Plant Breeding by Applying Next-Generation Artificial Intelligence
Harfouche, A. L., Jacobson, D. A., Kainer, D., Romero, J. C., Harfouche, A. H., Scarascia Mugnozza, G., Moshelion, M., Tuskan, G. A., Keurentjes, J. J. B., & Altman, A. (2019) · Trends in Biotechnology · doi:10.1016/j.tibtech.2019.05.007 · Cited by 292
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291
“Therefore, deep learning methods, including CNNs, often face the criticism that they are “blackbox” (Castelvecchi, 2016).”
Belenguer, L. (2022) · AI and Ethics · doi:10.1007/s43681-022-00138-8 · Cited by 289
Machine learning in clinical decision making
Adlung, L., Cohen, Y., Mor, U., & Elinav, E. (2021) · Med · doi:10.1016/j.medj.2021.04.006 · Cited by 282
“These are further discussed elsewhere.(10,178,179)”
Ma, Y., Xiang, Z., Du, Q., & Fan, W. (2018) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.12.008 · Cited by 284
Researching Mental Health Disorders in the Era of Social Media: Systematic Review
Wongkoblap, A., Vadillo, M. A., & Curcin, V. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7215 · Cited by 268
Cikes, M., Sanchez-Martinez, S., Claggett, B., Duchateau, N., Piella, G., Butakoff, C., Pouleur, A. C., Knappe, D., Biering-Sørensen, T., Kutyifa, V., et al. (2019) · European Journal of Heart Failure · doi:10.1002/ejhf.1333 · Cited by 266
“27 Furthermore, the ‘black-box’ nature of this methodology often provides results difficult to interpret.”
BackgroundInfluentialTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“Deploying such black-box models presents a serious dilemma in certain safety-critical domains, for instance, public health or finance [117].”
BackgroundQian, X., Pei, J., Zheng, H., Xie, X., Yan, L., Zhang, H., Han, C., Gao, X., Zhang, H., Zheng, W., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00711-2 · Cited by 253
“The black box nature of deep learning has been identified as an obstacle to establishing trust with human experts in clinical practice 45 .”
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
“Extensive increases in the amount of data and performance of computational devices have driven AI models to become more complex, to the point that neural networks have arguably become as opaque as the human brain [30].”
BackgroundHaque, A. B., Islam, A. K. M. N., & Mikalef, P. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 240
“…(Hengstler et al., 2016; Haque et al., 2021; Du and Xie, 2021); however, the working principle of AI systems is unclear as the machine-learning models used in different AI systems do not reveal enough information about the process through which the conclusion is derived (Castelvecchi, 2016).”
BackgroundBhargava, A., Bester, M., & Bolton, L. (2021) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-020-00153-8 · Cited by 235
“Participants appeared to lack knowledge on how RAIA works, how decisions are made, and what happens when they use RAIA; hence, it may be termed a “ black box ” (Castelvecchi 2016).”
BackgroundJoshi, A., Pradhan, B., Gite, S., & Chakraborty, S. (2023) · Remote Sensing · doi:10.3390/rs15082014 · Cited by 229
Fernandez, A., Herrera, F., Cordon, O., Jose del Jesus, M., & Marcelloni, F. (2019) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2881645 · Cited by 232
“One straightforward way for combining two of the most important concepts in ML (accuracy and interpretability/ explainability) in a natural way, to obtaining XAI models, is by means of the synergy between FRBSs and EAs.”
BackgroundInfluentialArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“The full tables describing the 21 tools for SLRs according to the 34 features are available in digital format at: https://angelosalatino. github.io/ai-slr/.”
Role of Artificial Intelligence in Shaping Consumer Demand in E-Commerce
Khrais, L. T. (2020) · Future Internet · doi:10.3390/fi12120226 · Cited by 228
Learning Molecular Representations for Medicinal Chemistry: Miniperspective
Chuang, K. V., Gunsalus, L. M., & Keiser, M. J. (2020) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.0c00385 · Cited by 222
Good proctor or “big brother”? Ethics of online exam supervision technologies
Coghlan, S., Miller, T., & Paterson, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00476-1 · Cited by 219
“( Kanchan, 2019) Compared to, perhaps, the plagiarism detection tool Turnitin, proctoring AI may strike users as highly opaque (Castelvecchi, 2016).”
BackgroundToward the third generation artificial intelligence
Zhang, B., Zhu, J., & Su, H. (2023) · Science China Information Sciences · doi:10.1007/s11432-021-3449-x · Cited by 217
“One of the major reasons is that most of the current deep learning uses function approximation algorithms, which in general are unequipped to understand the inner mechanism of human-like intelligence [76].”
BackgroundThe impact of intelligent cyber-physical systems on the decarbonization of energy
Inderwildi, O., Zhang, C., Wang, X., & Kraft, M. (2020) · Energy & Environmental Science · doi:10.1039/c9ee01919g · Cited by 216
How to make ecological models useful for environmental management
Schuwirth, N., Borgwardt, F., Domisch, S., Friedrichs, M., Kattwinkel, M., Kneis, D., Kuemmerlen, M., Langhans, S. D., Martínez-López, J., & Vermeiren, P. (2019) · Ecological Modelling · doi:10.1016/j.ecolmodel.2019.108784 · Cited by 210
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P. L., Zhu, Q., & Li, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.07107 · Cited by 213
Calibration of Pre-trained Transformers
Desai, S., & Durrett, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.21 · Cited by 209
Dennis, A. R., Lakhiwal, A., & Sachdeva, A. (2023) · Journal of Management Information Systems · doi:10.1080/07421222.2023.2196773 · Cited by 203
“AI-enabled technologies often produce concerns of information asymmetry because they are: “black-boxes” [20] that make it difficult to ascertain the information that an AI team member has and how it uses that information [123].”
BackgroundLiu, Q., & Xie, L. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008653 · Cited by 203
Machine learning and artificial intelligence in research and healthcare
Rubinger, L., Gazendam, A., Ekhtiari, S., & Bhandari, M. (2023) · Injury · doi:10.1016/j.injury.2022.01.046 · Cited by 203
Gregor, L., Lebehot, A. D., Kok, S., & Scheel Monteiro, P. M. (2019) · Geoscientific Model Development · doi:10.5194/gmd-12-5113-2019 · Cited by 202
“…is also somewhat diminished by the fact that little additional information that can be gained from highly non-linear models, with the exception of basic diagnostics such as feature-variable importance (see Figure S11) from decision-tree-based approaches (Pedregosa et al. 2012; Castelvecchi, 2016).”
Research on Artificial Intelligence Enhancing Internet of Things Security: A Survey
Wu, H., Han, H., Wang, X., & Sun, S. (2020) · IEEE Access · doi:10.1109/access.2020.3018170 · Cited by 201
“3) POOR INTERPRETABILITY / LOW TRANSPARENCY / BLACK BOX Deep neural network models are like black boxes because we have no way of knowing how they draw conclusions by manipulating parameters and input data [94].”
BackgroundPost-hoc vs ante-hoc explanations: xAI design guidelines for data scientists
Retzlaff, C. O., Angerschmid, A., Saranti, A., Schneeberger, D., Röttger, R., Müller, H., & Holzinger, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101243 · Cited by 194
“Due to the growing challenge posed by the so-called ‘‘black box’’ problem (Castelvecchi, 2016) in increasingly complex models of Artificial Intelligence (AI) across various domains, there has been a significant global interest in explainable Artificial Intelligence (xAI) (Con-falonieri, Coba et…”
Artificial intelligence for precision oncology: beyond patient stratification
Azuaje, F. (2019) · npj Precision Oncology · doi:10.1038/s41698-019-0078-1 · Cited by 200
“39 Interpretability can be achieved at different levels of data processing and abstraction, and may not be necessarily constrained to speci fi c models.”
BackgroundWang, Y. (2021) · Journal of Educational Administration · doi:10.1108/jea-10-2020-0216 · Cited by 196
Li, F., Chen, H., Liu, Z., Zhang, X.-D., Jiang, M.-S., Wu, Z.-Z., & Zhou, K.-Q. (2019) · Biomedical Optics Express · doi:10.1364/boe.10.006204 · Cited by 193
Diogo, V. S., Ferreira, H. A., Prata, D., & Initiative, F. T. A. D. N. (2022) · Alzheimer's Research & Therapy · doi:10.1186/s13195-022-01047-y · Cited by 184
“Most previously published classifiers largely lack interpretability and do not provide any insight into how the classification decision was reached [11, 17].”
BackgroundA systematic review of personal thermal comfort models
Arakawa Martins, L., Soebarto, V., & Williamson, T. (2022) · Building and Environment · doi:10.1016/j.buildenv.2021.108502 · Cited by 187
Xu, H., & Shuttleworth, K. M. J. (2024) · Intelligent Medicine · doi:10.1016/j.imed.2023.08.001 · Cited by 196
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185
Toward a unified framework for interpreting machine-learning models in neuroimaging
Kohoutová, L., Heo, J., Cha, S., Lee, S., Moon, T., Wager, T. D., & Woo, C.-W. (2020) · Nature Protocols · doi:10.1038/s41596-019-0289-5 · Cited by 176
Open-source QSAR models for pKa prediction using multiple machine learning approaches
Mansouri, K., Cariello, N. F., Korotcov, A., Tkachenko, V., Grulke, C. M., Sprankle, C. S., Allen, D., Casey, W. M., Kleinstreuer, N. C., & Williams, A. J. (2019) · Journal of Cheminformatics · doi:10.1186/s13321-019-0384-1 · Cited by 177
“While DNNs have shown remarkable performance in many areas, in many cases they remain a black box [65].”
BackgroundAlabi, R. O., Youssef, O., Pirinen, M., Elmusrati, M., Mäkitie, A. A., Leivo, I., & Almangush, A. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102060 · Cited by 174
Giuliani, M., Lamontagne, J. R., Reed, P. M., & Castelletti, A. (2021) · Water Resources Research · doi:10.1029/2021wr029927 · Cited by 167
Eitel, F., Soehler, E., Bellmann-Strobl, J., Brandt, A. U., Ruprecht, K., Giess, R. M., Kuchling, J., Asseyer, S., Weygandt, M., Haynes, J.-D., et al. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.102003 · Cited by 171
“Deep learning methods are often criticized for their lack of interpretability and over the last years much research has focused on improving the interpretability of neural networks (Castelvecchi, 2016; Montavon et al., 2018; Lapuschkin et al., 2019).”
BackgroundUnto the breach: What the COVID-19 pandemic exposes about digitalization
Faraj, S., Renno, W., & Bhardwaj, A. (2021) · Information and Organization · doi:10.1016/j.infoandorg.2021.100337 · Cited by 171
“For example, an early version of a self-driving car algorithm turned out to be relying on the presence of sidewalk grass to identify the boundary of the road, initially resulting in unexpected and dangerous swerving when the grass ended (Castelvecchi, 2016).”
BackgroundLukyanenko, R., Maass, W., & Storey, V. C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00605-4 · Cited by 171
“The opacity of such models undermines the ability to understand and explain how and why such models make their decisions (Adadi & Berrada, 2018; Castelvecchi, 2016; D Gunning, 2016; Mueller et al., 2019; Storey et al., 2022).”
Harnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 393
“Although the black box is not entirely untrustworthy [16], the results of formal study show that all participants demanded further clarification from ChatGPT to enhance the transparency of the results.”
Metadata
- Authors
- Castelvecchi, D.
- Corresponding author
- Castelvecchi, Davide
- Venue
- Nature
- Published
- 2016-10
- Volume
- 538
- Issue
- 7623
- Pages
- 20-23
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2528491735
- Cited by
- 1,773
- References
- 6
- Citations by year
- 2026: 195 · 2025: 322 · 2024: 276 · 2023: 244 · 2022: 217 · 2021: 214 · 2020: 148 · 2019: 84 · 2018: 54 · 2017: 18 · 2016: 1
- FWCI
- 42.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-sa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion, cc-by-sa)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- artificial intelligence, machine learning, interpretability, explainability, trust in AI
- MeSH terms
- Machine Learning, Astronomy, Creativity, Humans, Physics, Robotics, Thinking, Neural Networks, Computer, Uncertainty, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.