Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 401–500 of 1,007
Machine Learning Applications in the Neuro ICU: A Solution to Big Data Mayhem?
Chaudhry, F., Hunt, R., Hariharan, P., Anand, S. K., Sanjay, S., Kjoller, E. E., Bartlett, C. M., Johnson, K. W., Levy, P. D., Noushmehr, H., et al. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.554633 · Cited by 32
“For example: artificial neural networks (ANN) tends to lose predictive power with smaller datasets (11).”
BackgroundHenckaerts, R., Antonio, K., & Côté, M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117230 · Cited by 39
Attack-agnostic Adversarial Detection on Medical Data Using Explainable Machine Learning
Watson, M., & Moubayed, N. A. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412560 · Cited by 38
Explainable Natural Language Processing
Søgaard, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01118ed1v01y202107hlt051 · Cited by 38
ExMed: An AI Tool for Experimenting Explainable AI Techniques on Medical Data Analytics
Kapcia, M., Eshkiki, H., Duell, J., Fan, X., Zhou, S., & Mora, B. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00134 · Cited by 36
“A goal of enabling explainability in AI systems “is to ensure algorithmic predictions and any input data triggering those predictions can be explained” [3].”
Correa-Jullian, C., & Groth, K. (2022) · International journal of hydrogen energy · doi:10.1016/j.ijhydene.2022.04.048 · Cited by 29
Sentimental wildfire: a social-physics machine learning model for wildfire nowcasting
Lever, J., & Arcucci, R. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00174-8 · Cited by 34
Aldeia, G. S. I., & de França, F. O. (2022) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-022-09435-x · Cited by 38
Islam, M. T., Syfullah, M. K., Rashed, M. G., & Das, D. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02242-z · Cited by 37
“Based on [16, 17], the categorization of XAI can be organized into different aspects.”
Raufi, B., Finnegan, C., & Longo, L. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63803-9_20 · Cited by 35
Alshkeili, H. M. H. A., Almheiri, S. J., & Khan, M. A. (2025) · AI · doi:10.3390/ai6060117 · Cited by 41
“The researchers in [38] established two main interpretability approaches: local interpretations and global explanations.”
Explainable AI for trustworthy intelligent process monitoring
Johannssen, A., Qiu, P., Yeganeh, A., & Chukhrova, N. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111407 · Cited by 43
Apata, O., Munda, J. L., & Migabo, E. (2026) · Energies · doi:10.3390/en19020536 · Cited by 37
Employing Explainable AI to Optimize the Return Target Function of a Loan Portfolio
Gramespacher, T., & Posth, J.-A. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.693022 · Cited by 31
“In many circumstances, and especially in fi nance, users of ML models would not or could not simply accept the output of a black box model, but would require at least some level of interpretability or explainability of the model before it could be used (Ribeiro et al., 2016; Carvalho et al., 2019).”
BackgroundMethodAdViCE: Aggregated Visual Counterfactual Explanations for Machine Learning Model Validation
Gómez, Ó., Holter, S., Yuan, J., & Bertini, E. (2021) · Visualization · doi:10.1109/vis49827.2021.9623271 · Cited by 38
DiCE4EL: Interpreting Process Predictions using a Milestone-Aware Counterfactual Approach
Hsieh, C., Moreira, C., & Ouyang, C. (2021) · International Conference on Process Mining (ICPM) · doi:10.1109/icpm53251.2021.9576881 · Cited by 32
Advances in XAI: Explanation Interfaces in Healthcare
Manresa-Yee, C., Roig-Maimó, M. F., Ramis, S., & Mas-Sansó, R. (2021) · Intelligent systems reference library · doi:10.1007/978-3-030-83620-7_15 · Cited by 41
Zaunseder, E., Haupt, S., Mütze, U., Garbade, S. F., Kölker, S., & Heuveline, V. (2022) · JIMD Reports · doi:10.1002/jmd2.12285 · Cited by 35
“33 Decision trees 7,13 – 15, 19 , 24 ,”
MethodA Meta Survey of Quality Evaluation Criteria in Explanation Methods
Löfström, H., Hammar, K., & Johansson, U. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-07481-3_7 · Cited by 46
“The criteria are considered by some authors more of desiderata and are therefore somewhat lacking in method description [11].”
BackgroundInfluentialGwak, M., Kim, M. S., Yun, J. P., & Park, P. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3207758 · Cited by 35
“Althoughthedecisioncriteriahavebeenaddressedinthe manyexistingXAIstudies,therobustnessofthevibration classificationmodelshasbeenpoorlyattemptedtoexplain[13], [14],[21],[22],[23].”
Llama 2: Early Adopters' Utilization of Meta's New Open-Source Pretrained ModelPreprint
Roumeliotis, K. I., Tselikas, N. D., & Nasiopoulos, D. Κ. (2023) · Preprints.org · doi:10.20944/preprints202307.2142.v2 · Cited by 51
Calibrated explanations: With uncertainty information and counterfactuals
Löfström, H., Löfström, T., Johansson, U., & Sönströd, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123154 · Cited by 43
Geethanjali, K. S., & Umashankar, N. (2025) · International Conference on Emerging Systems and Intelligent Computing (ESIC) · doi:10.1109/esic64052.2025.10962568 · Cited by 30
“While XAI offers promising solutions, several challenges remain in its integration into educational systems [14][15]: • Scalability: It is still very difficult to create XAI solutions that scale across various educational contexts, such as various subjects, grade levels, and cultural settings.”
Comprehensive Perspective for Lung Cancer Characterisation Based on AI Solutions Using CT Images
Pereira, T., Freitas, C., Costa, J. L., Morgado, J., Silva, F., Negrão, E., de Lima, B. F., da Silva, M. C., Madureira, A. J., Ramos, I. M., et al. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm10010118 · Cited by 34
Coalitional Strategies for Efficient Individual Prediction Explanation
Ferrettini, G., Escriva, E., Aligon, J., Excoffier, J.-B., & Soulé-Dupuy, C. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10141-9 · Cited by 35
“That the approach is used by a large part of the literature to understand how a model works (Adadi and Berrada 2018; Carvalho et al. 2019).”
BackgroundAppropriate use of machine learning in healthcare
Ozaydin, B., Berner, E. S., & Cimino, J. J. (2021) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2021.100041 · Cited by 33
Perennial ryegrass biomass retrieval through multispectral UAV data
de Alckmin, G. T., Lucieer, A., Rawnsley, R., & Kooistra, L. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106574 · Cited by 33
García-Vicente, C., Chushig-Muzo, D., Mora-Jiménez, I., Fabelo, H. A., Gram, I. T., Løchen, M., Granja, C., & Soguero-Ruíz, C. (2023) · Applied Sciences · doi:10.3390/app13074119 · Cited by 36
Urban Air Quality Classification Using Machine Learning Approach to Enhance Environmental Monitoring
Idroes, G. M., Noviandy, T. R., Maulana, A., Zahriah, Z., Suhendrayatna, S., Suhartono, E., Khairan, K., Kusumo, F., Helwani, Z., & Rahman, S. A. (2023) · Leuser Journal of Environmental Studies · doi:10.60084/ljes.v1i2.99 · Cited by 31
MLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 39
“Causality: Interpretability gives a better understanding of the model decisions and the features and hence can help to identify causal relationships of the features [46].”
Dhanekula, A., & Miah, M. R. (2022) · American Journal of Interdisciplinary Studies · doi:10.63125/707veh84 · Cited by 30
NBDT: Neural-Backed Decision Tree
Wan, A., Dunlap, L., Ho, D. E., Yin, J., Lee, S., Petryk, S., Bargal, S. A., & Gonzalez, J. E. (2021) · Cited by 58
Wong, L. J., Clark, W. H., Flowers, B., Buehrer, R. M., Michaels, A. J., & Headley, W. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00432 · Cited by 32
“Additional challenges to ML interpretations include, but are not limited to [194]:”
BackgroundEvaluating the Correctness of Explainable AI Algorithms for ClassificationPreprint
Yalcin, M. O., Fan, X., & Liu, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09740 · Cited by 35
“A survey on machine learning interpretability is presented in [4].”
BackgroundConstrained Prescriptive Trees via Column Generation
Subramanian, S., Sun, W., Drissi, Y., & Ettl, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20384 · Cited by 38
“There has been a surge of interest in making ML models more interpretable in the recent years (e.g., Ribeiro, Singh, and Guestrin 2016, Lundberg and Lee 2017, Guidotti et al. 2018, Carvalho, Pereira, and Cardoso 2019).”
Explainable machine learning models with privacy
Bozorgpanah, A., & Torra, V. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00315-2 · Cited by 36
Ivanisenko, T., Сергеевич, Д. П., & Иванисенко, В. А. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252111811 · Cited by 40
Ali, H., Pham, D.-T., Alam, S., Schultz, M., Li, M. Z., Wang, Y., Itoh, E., & Dương, V. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103256 · Cited by 32
Xenochristou, M., Hutton, C., Hofman, J. A. M. H., & Kapelan, Z. S. (2021) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.0001325 · Cited by 30
“…of machine learning models is a topic with increasing popularity as more methods are developed (Doshi-Velez and Kim, 2017; Adadi and Berrada, 2018; Carvalho et al., 2019; Molnar, 2019a) and find use in different fields, particularly in medical applications (Berk et al., 2016; Choi, 2018; Cremona…”
MethodExplainability Metrics of Deep Convolutional Networks for Photoplethysmography Quality Assessment
Zhang, O. R., Ding, C., Pereira, T., Xiao, R., Gadhoumi, K., Meisel, K. M., Lee, R. J., Chen, Y., & Hu, X. (2021) · IEEE Access · doi:10.1109/access.2021.3054613 · Cited by 32
“Interpretability techniques for machine learning can be broken up into two classes: local explanations and global explanations [12], [13].”
BackgroundIzima, O., de Fréin, R., & Malik, A. (2021) · Electronics · doi:10.3390/electronics10222851 · Cited by 31
SoK: Explainable Machine Learning in Adversarial Environments
Noppel, M., & Wressnegger, C. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00021 · Cited by 49
“We emphasize that we do not attempt to summarize the field of XAI and refer the reader to surveys on the topic [2, 31, 36, 39, 45, 46, 67, 109, 155, 199].”
BackgroundHuman-Driven FOL Explanations of Deep Learning
Ciravegna, G., Giannini, F., Gori, M., Maggini, M., & Melacci, S. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/309 · Cited by 38
Interpretable Poverty Mapping using Social Media Data, Satellite Images, and Geospatial InformationPreprint
Ledesma, C., Garonita, O. L., Flores, L. J., Tingzon, I., & Dalisay, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13563 · Cited by 35
Chen, Y., Wang, G., Zhu, Y., Xie, C., & Uddin, G. S. (2024) · European Journal of Finance · doi:10.1080/1351847x.2024.2358940 · Cited by 30
“ML models are like ‘black boxes’ that severely hinder people from interpreting their predictions or decisions (Carvalho, Pereira, and Cardoso 2019; Minh et al.2022 ; Zhou et al.2021 ).”
How do ML practitioners perceive explainability? an interview study of practices and challenges
Habiba, U.-E., Habib, M. K., Bogner, J., Fritzsch, J., & Wagner, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10565-2 · Cited by 37
“Additionally, existing model interpretability methods (Carvalho et al. 2019) may not cater explicitly to end-users (Suresh et al. 2021), requiring more user-friendly explanations.”
Pre-Training Autoencoder for Lung Nodule Malignancy Assessment Using CT Images
Silva, F., Pereira, T., Frade, J., Mendes, J., Freitas, C., Hespanhol, V. P., Costa, J. L., Cunha, A. M. T. D. S., & Oliveira, H. P. (2020) · Applied Sciences · doi:10.3390/app10217837 · Cited by 33
Kang, S., Jin, R., Deng, X., & Kenett, R. S. (2021) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-021-01817-9 · Cited by 33
“In the meantime, a known limitation of such AL algorithms is that they are mostly black-box models in nature with highly nonlinear model structure, thus not interpretable to human (Carvalho et al. 2019).”
BackgroundChaquet-Ulldemolins, J., Gimeno-Blanes, F.-J., Moral‐Rubio, S., Muñoz-Romero, S., & Rojo‐Álvarez, J. L. (2022) · Applied Sciences · doi:10.3390/app12073328 · Cited by 34
“In like fashion, the decision models need to be an easily understandable, or in other words, they need to allow us to check which attributes are necessary to produce explanations that are comprehensible [11].”
BackgroundMethodXie, C., Zhang, J., Zhu, Y., Xiong, B., & Wang, G. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108874 · Cited by 34
Charlton, C. E., Poon, M. T. C., Brennan, P. M., & Fleuriot, J. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107482 · Cited by 33
Enabling causality learning in smart factories with hierarchical digital twins
Lippi, M., Martinelli, M., Picone, M., & Zambonelli, F. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103892 · Cited by 35
Corti, L., Oltmans, R., Jung, J., Balayn, A., Wijsenbeek, M. S., & Yang, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642551 · Cited by 43
“7 First, patients are encouraged to adopt healthy lifestyles (e.g., stop smoking or exercise regularly).”
InfluentialSpeith, T., Crook, B., Mann, S., Schomäcker, A., & Langer, M. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09769-3 · Cited by 34
“Explainability is relevant for such a system, for instance, to ensure the system’s outputs do not discriminate unfairly against any protected groups (e.g., women, people of color, LGBTQIA*; Carvalho et al., 2019; Langer & König, 2023).”
A novel classification algorithm for customer churn prediction based on hybrid Ensemble-Fusion model
He, C., & Ding, C. H. Q. (2024) · Scientific Reports · doi:10.1038/s41598-024-71168-x · Cited by 27
“Then this paper validates the proposed churn prediction algorithm experimentally and evaluates the robustness of the algorithm by using evaluation metrics such as precision, recall, accuracy, F1-score, and AUC.”
BackgroundInfluentialMineral prospectivity mapping susceptibility evaluation based on interpretable ensemble learning
Yu, Z., Li, B., & Wang, X. (2024) · Ore Geology Reviews · doi:10.1016/j.oregeorev.2024.106248 · Cited by 30
Ndao, M. L., Youness, G., Niang, N., & Saporta, G. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110144 · Cited by 27
de Oliveira, A. C., Azevedo, J. P. C., Ruback, L., Moreira, R., Teixeira, S. S., & Teles, A. S. (2025) · IEEE Access · doi:10.1109/access.2025.3556245 · Cited by 28
“However, each has a distinct definition [19], [89].”
Rai, N. K., Sen, O., & Udaykumar, H. S. (2020) · Journal of Applied Physics · doi:10.1063/5.0010492 · Cited by 27
Nieto-del-Amor, F., Prats-Boluda, G., Martinez-de-Juan, J. L., Diaz‐Martinez, A., Ortiz, I. R. M., Diago, V., & Ye-Lin, Y. (2021) · Sensors · doi:10.3390/s21103350 · Cited by 29
Wong, L. J., Clark, W. H., Flowers, B., Buehrer, R. M., Headley, W. C., & Michaels, A. J. (2021) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2021.3112939 · Cited by 29
Computer Vision and Machine Learning for Tuna and Salmon Meat Classification
Medeiros, E. C., Almeida, L. M., & Filho, J. G. D. A. T. (2021) · Informatics · doi:10.3390/informatics8040070 · Cited by 31
“We evaluated the AutoML models to classify the freshness levels of tuna and salmon samples through the metrics: accuracy [50], receiver operating characteristic (ROC) curve [51], precision [52], recall [53], f1-score [54], and confusion matrix (CM) [55].”
BackgroundMethodHealthcare analytics: A techno-functional perspective
Singha, S., Arha, H., & Kar, A. K. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122908 · Cited by 31
Exploring explainable AI features in the vocal biomarkers of lung disease
Chen, Z., Liang, N., Li, H., Zhang, H., Li, H., Yan, L., Hu, Z., Chen, Y., Zhang, Y., Wang, Y., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108844 · Cited by 26
Ma, K., Shen, C., Xu, Z., & He, D. (2024) · Journal of Geographical Sciences · doi:10.1007/s11442-024-2235-x · Cited by 28
“On the other hand, there has been rising concern over the interpretability of ML models due to their strength and complexity (Carvalho et al ., 2019; Linardatos et al ., 2020), and some progress has been made to reveal potential patterns by constraining the training process with physical…”
Counterfactual Explanation of Machine Learning Survival Models
Kovalev, M. S., Utkin, L. V., Coolen, F. P. A., & Konstantinov, A. V. (2021) · Informatica · doi:10.15388/21-infor468 · Cited by 32
“ver, the corresponding methods are computationally complex when samples are of the high dimensionality. A lot of explanation methods, their analysis, and critical review can be found in survey papers [1, 3, 10, 25, 55, 75]. Most explanation methods deal with the point-valued results produced by explainable black-box m…”
BackgroundSuresh, H., Lewis, K. M., Guttag, J., & Satyanarayan, A. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511160 · Cited by 20
“A major area of focus has been on developing methodologies for computing such explanations [14].”
BackgroundSykes, A. L., Silva, G. S., Holtkamp, D., Mauch, B. W., Osemeke, O., Linhares, D. C. L., & Machado, G. (2021) · Transboundary and Emerging Diseases · doi:10.1111/tbed.14369 · Cited by 30
A social evaluation of the perceived goodness of explainability in machine learning
Wanner, J., Herm, L.-V., Heinrich, K., & Janiesch, C. (2021) · Journal of Business Analytics · doi:10.1080/2573234x.2021.1952913 · Cited by 27
“Preliminary work developed and adapted these metrics for XAI research (e.g., reviewed in Carvalho et al., 2019; Mohseni et al., 2018).”
BackgroundMethodA Survey on Explainable Anomaly Detection for Industrial Internet of Things
Huang, Z., & Wu, Y. (2022) · IEEE Conference on Dependable and Secure Computing (DSC) · doi:10.1109/dsc54232.2022.9888874 · Cited by 33
“Using the term of explainable machine learning, Diogo et al. [23] gave an overview of machine learning interpretability regarding methods and metrics.”
Utkin, L. V., & Konstantinov, A. V. (2022) · Algorithms · doi:10.3390/a15110431 · Cited by 29
Elkhawaga, G., Elzeki, O. M., Abu-Elkheir, M., & Reichert, M. (2023) · Electronics · doi:10.3390/electronics12071670 · Cited by 31
What is missing in the study of emotion expression?
Straulino, E., Scarpazza, C., & Sartori, L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1158136 · Cited by 32
Machine learning for fast development of advanced energy materials
Farhadi, B., You, J., Zheng, D., Liu, L., Wu, S., Li, J., Li, Z., Wang, K., & Liu, S. F. (2023) · Next Materials · doi:10.1016/j.nxmate.2023.100025 · Cited by 27
Through the looking glass: evaluating post hoc explanations using transparent models
Velmurugan, M., Ouyang, C., Sindhgatta, R., & Moreira, C. (2023) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00445-1 · Cited by 27
“This is necessary to ensure system quality and to facilitate informed human decision-making [8].”
Mirzaei, S., Mao, H., Al-Nima, R. R. O., & Woo, W. L. (2023) · Information · doi:10.3390/info15010004 · Cited by 25
“A study conducted by Carvalho et al. [6] showed that when two models contribute similar features to arrive at a prediction with consistent explanations, they exhibit a high degree of consistency, signifying a robust explanatory characteristic.”
BackgroundInfluentialCold-Start Promotional Sales Forecasting Through Gradient Boosted-Based Contrastive Explanations
Aguilar-Palacios, C., Muñoz-Romero, S., & Rojo‐Álvarez, J. L. (2020) · IEEE Access · doi:10.1109/access.2020.3012032 · Cited by 34
“works in the research community [15]–[19] aiming to clarify and facilitate the understanding of ML and Deep Learning models.”
BackgroundTian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398 · Cited by 31
Wang, Y., Yao, Q., Zhang, Q., Zhang, H., Lu, Y., Fan, Q., Jiang, N., & Yu, W. (2022) · Nuclear Engineering and Technology · doi:10.1016/j.net.2022.08.011 · Cited by 25
AutoXAI: A Framework to Automatically Select the Most Adapted XAI Solution
Cugny, R., Aligon, J., Chevalier, M., Roman‐Jimenez, G., & Teste, O. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557247 · Cited by 44
“As mentioned in Section 1, many XAI solutions now exist and different taxonomies have been proposed such as Barredo Arrieta et al. [2020], Carvalho et al. [2019], Adadi and Berrada [2018], Gilpin et al. [2018], Nauta et al. [2022].”
BackgroundMethodInfluentialInterpretability for Multimodal Emotion Recognition using Concept Activation Vectors
Asokan, A. R., Kumar, N., Ragam, A. V., & Shylaja, S. S. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892315 · Cited by 32
“e, the internal reasoning process of these models are hidden from the user, which can prove to be a pitfall that prevents humans from verifying the decisions made by these black box models [1].”
BackgroundHabiba, U.-E., Bogner, J., & Wagner, S. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00038 · Cited by 35
“In particular, explainability and interpretability appear frequently in the literature, and many researchers use them interchangeably [16] as no specific mathematical definitions for explainability or interpretability exist, nor have they been quantified by any metric [17].”
BackgroundInterpretable Diabetic Retinopathy Diagnosis Based on Biomarker Activation Map
Zang, P., Hormel, T. T., Wang, J., Guo, Y., Bailey, S. T., Flaxel, C. J., Huang, D., Hwang, T. S., & Jia, Y. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3290541 · Cited by 27
“1 in the Supplementary) [28], [29], [30], which is distinct from DR diagnosis in several regards and consequently poses many challenges in presenting clinically meaningful attention maps.”
BackgroundSequential stacking link prediction algorithms for temporal networks
He, X., Ghasemian, A., Lee, E., Clauset, A. J., & Mucha, P. J. (2024) · Nature Communications · doi:10.1038/s41467-024-45598-0 · Cited by 33
Wang, L., Yang, J., Wu, S., Hu, L., Ge, Y., & Du, Z. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103746 · Cited by 47
FDKT: Towards an Interpretable Deep Knowledge Tracing via Fuzzy Reasoning
Liu, F., Bu, C., Zhang, H., Wu, L., Yu, K., & Hu, X. (2024) · ACM Transactions on Information Systems · doi:10.1145/3656167 · Cited by 42
“Being able to explain the reasons why the model was able to achieve good prediction performance in an interpretable KT model is as crucial as achieving desirable performance [17].”
BackgroundA Survey of Trustworthy Representation Learning Across Domains
Zhu, R., Guo, D., Qi, D., Chu, Z., Yu, X., & Li, S. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3657301 · Cited by 31
Improving evidence-based assessment of players using serious games
Alonso‐Fernández, C., Freire, M., Martínez‐Ortiz, I., & Fernández‐Manjón, B. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2021.101583 · Cited by 28
“With the resulting GLA variables, interpretable prediction models (Carvalho, Pereira, & Cardoso, 2019) can be tested during the game validation phase and, in the deployment phase, used to assess players.”
MethodLiu, M., Wen, Z., & Su, H. (2024) · Engineering Structures · doi:10.1016/j.engstruct.2024.118845 · Cited by 25
Jhanjhi, N. Z. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-91005-0_43 · Cited by 31
Introduction to Interpretability and Explainability
Kamath, U., & Liu, J. (2021) · Springer eBooks · doi:10.1007/978-3-030-83356-5_1 · Cited by 28
Nieto-del-Amor, F., Prats-Boluda, G., Garcia‐Casado, J., Diaz‐Martinez, A., Diago-Almela, V. J., Ortiz, I. R. M., Hao, D., & Ye-Lin, Y. (2022) · Sensors · doi:10.3390/s22145098 · Cited by 27
“Obstetricians find the prediction results difficult to interpret and hard to trust, since these algorithms achieve good performance even when the input features are highly overlapped between the target classes [19,23].”
BackgroundAccess to online learning: Machine learning analysis from a social justice perspective
McIntyre, N. A. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11280-5 · Cited by 24
“Ultimately, interpretability and usefulness of a machine learning model is domain specific (Carvalho et al., 2019).”
BackgroundCharacterization of Synthetic Health Data Using Rule-Based Artificial Intelligence Models
Lenatti, M., Paglialonga, A., Orani, V., Ferretti, M., & Mongelli, M. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3236722 · Cited by 25
“EXplainable AI (XAI) techniques could help assess if, and to what extent synthetic data maintain input-output relationships similar to those found in real data [15], [16].”
ResultZhai, S., Chen, K., Yang, L., Li, Z., Yu, T., Chen, L., & Zhu, H. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.170232 · Cited by 24
Machine learning approaches to natural fiber composites: A review of methodologies and applications
Palanisamy, S., Ayrılmış, N., Sureshkumar, K., Santulli, C., Khan, T., Junaedi, H., & Sebaey, T. A. (2025) · BioResources · doi:10.15376/biores.20.1.palanisamy · Cited by 32
“Data scientists play a crucial role in processing and analyzing vast datasets, implementing advanced ML algorithms, and developing interpretable models that align with industry standards and regulatory requirements (Carvalho et al. 2019; Shah 2021).”
Llama 2: Early Adopters' Utilization of Meta's New Open-Source Pretrained ModelPreprint
Roumeliotis, K. I., Tselikas, N. D., & Nasiopoulos, D. Κ. (2023) · Preprints.org · doi:10.20944/preprints202307.2142.v1 · Cited by 40
Guo, Z., Feng, C., Liu, Y., & Liu, Q. (2024) · Agricultural Water Management · doi:10.1016/j.agwat.2024.109157 · Cited by 28
Madeyski, L., & Stradowski, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112360 · Cited by 26
Roy, D. K., Sarkar, T. K., Munmun, T. H., Paul, C. R., & Datta, B. (2025) · Discover Water · doi:10.1007/s43832-025-00207-z · Cited by 23
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.