Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 301–400 of 1,007
Explainable artificial intelligence models in intrusion detection systems
Al, S., & Sağıroğlu, Ş. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110145 · Cited by 71
Interpretable Machine Learning for COVID-19: An Empirical Study on Severity Prediction Task
Wu, H., Ruan, W., Wang, J., Zheng, D., Liu, B., Geng, Y., Chai, X., Chen, J., Li, K., Li, S., et al. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3092698 · Cited by 52
“there are two types of indicators for assessment and comparison of explanations: qualitative and quantitative indicators [23].”
BackgroundIncorporating Residual and Normalization Layers into Analysis of Masked Language Models
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.373 · Cited by 46
“As current neural-based models have an end-to-end, black-box nature, existing studies have adopted several strategies to interpret their inner workings (Carvalho et al., 2019; Rogers et al., 2020; Braşoveanu and Andonie, 2020).”
Mehrvar, S., Himmel, L. E., Babburi, P., Goldberg, A. L., Guffroy, M., Janardhan, K. S., Krempley, A. L., & Bawa, B. (2021) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_36_21 · Cited by 49
The Role of Human Knowledge in Explainable AI
Tocchetti, A., & Brambilla, M. (2022) · Data · doi:10.3390/data7070093 · Cited by 55
“For a complete and detailed summary of the state-of-the-art explainability, we advise the reader to refer to [2,9,10,17,18].”
BackgroundXia, T., Han, J., & Mascolo, C. (2022) · Experimental Biology and Medicine · doi:10.1177/15353702221115428 · Cited by 71
“81 On the other side, attention mechanism could be a plausible option to be incorporated in deep neural networks so that the significance of different spectrogram segments can be traced.”
BackgroundHuang, J., Wu, X., Ling, S., Li, X., Wu, Y., Peng, L., & He, Z. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-23732-z · Cited by 46
Explainable Artificial Intelligence in Cybersecurity: A Brief Review
Hariharan, S., Velicheti, A., Anagha, A. S., Thomas, C., & Balakrishnan, N. (2021) · doi:10.1109/isea-isap54304.2021.9689765 · Cited by 54
Chaquet-Ulldemolins, J., Gimeno-Blanes, F.-J., Moral‐Rubio, S., Muñoz-Romero, S., & Rojo‐Álvarez, J. L. (2022) · Applied Sciences · doi:10.3390/app12083856 · Cited by 61
“To cope with it, and according to existing literature, financial institutions rely on using simple interpretable models, such as decision trees [28] or linear models [12].”
Giering, S. L. C., Culverhouse, P. F., Johns, D. G., McQuatters‐Gollop, A., & Pitois, S. G. (2022) · Frontiers in Marine Science · doi:10.3389/fmars.2022.986206 · Cited by 47
“XAI has gained attention only in the past decade (Carvalho et al., 2019) but is now considered critical for the widespread adoption of AI (e.”
BackgroundSpatial Decision Support Systems with Automated Machine Learning: A Review
Wen, R., & Li, S. (2022) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12010012 · Cited by 49
Sahlaoui, H., Alaoui, E. A. A., Agoujil, S., & Nayyar, A. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12007-w · Cited by 48
“(Carvalho, 2019) It was evident from these uses that machine learning models have a substantial influence on students’ professional achievement; hence, it is crucial that the model be well understood and devoid of bias.”
Survey on ontology-based explainable AI in manufacturing
Naqvi, M. R., Elmhadhbi, L., Sarkar, A., Archimède, B., & Karray, M. H. (2024) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02304-z · Cited by 49
“In cognitive XAI, they consider explanations as to the Intentional stance of the agent (Carvalho et al., 2019).”
InfluentialNicolson, A., Bradburn, E., Gal, Y., Papageorghiou, A. T., & Noble, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02023-0 · Cited by 57
Variational Autoencoder-Based Vehicle Trajectory Prediction with an Interpretable Latent Space
Neumeier, M., Botsch, M., Tollkühn, A., & Berberich, T. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9565120 · Cited by 46
Fuzzy clustering decomposition of genetic algorithm-based instance selection for regression problems
Kordos, M., Blachnik, M., & Scherer, R. (2021) · Information Sciences · doi:10.1016/j.ins.2021.12.016 · Cited by 46
Developing a Fidelity Evaluation Approach for Interpretable Machine LearningPreprint
Velmurugan, M., Ouyang, C., Moreira, C., & Sindhgatta, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08492 · Cited by 53
“As such, the research theme of XAI has arisen in order to provide the ability to interpret the decision making of black box prediction models, in order to ensure system quality and facilitate informed decision-making (Carvalho et al, 2019).”
BackgroundMultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents
Feng, S., Patel, S. S., Wan, H., & Joshi, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.498 · Cited by 44
“To train a dialogue system to be able to provide Fine-grained grounding information can be an important step for improving the interpretability and trustworthiness of neural-model-based conversational systems (Carvalho et al., 2019).”
BackgroundInterpretable Machine Learning for Prediction of Post-Fire Self-Healing of Concrete
Rajczakowska, M., Szeląg, M., Habermehl-Cwirzen, K., Hedlund, H., & Ćwirzeń, A. (2023) · Materials · doi:10.3390/ma16031273 · Cited by 46
“One of the significant drawbacks of ML modeling is a lack of interpretability [29] leading to their limited applicability.”
BackgroundFactors influencing trust in algorithmic decision-making: an indirect scenario-based experiment
Marmolejo‐Ramos, F., Marrone, R. L., Korolkiewicz, M. W., Gabriel, F. C., Siemens, G., Joksimovíc, S., Yamada, Y., Mori, Y., Rahwan, T., Sahakyan, M., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1465605 · Cited by 52
Zhang, S., Zhang, J., Li, X., Du, X., Zhao, T., Hou, Q., & Jin, X. (2022) · Ecological Indicators · doi:10.1016/j.ecolind.2022.108533 · Cited by 49
A new interpretable streamflow prediction approach based on SWAT-BiLSTM and SHAP
Huang, F., & Zhang, X. (2024) · Environmental Science and Pollution Research · doi:10.1007/s11356-024-32725-z · Cited by 46
“Interpretability has gradually become an important research direction in the field of machine learning in recent years (Carvalho et al. 2019).”
Brain tumor detection using proper orthogonal decomposition integrated with deep learning networks
Appiah, R., Pulletikurthi, V., Esquivel-Puentes, H. A., Cabrera, C., Hasan, N. I., Dharmarathne, S., Gomez, L. J., & Castillo, L. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108167 · Cited by 52
Uncertainty in XAI: Human Perception and Modeling Approaches
Chiaburu, T., Haußer, F., & Bießmann, F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020055 · Cited by 53
Insights into radiomics: a comprehensive review for beginners
Mariotti, F., Agostini, A., Borgheresi, A., Marchegiani, M., Zannotti, A., Giacomelli, G., Pierpaoli, L., Tola, E., Galiffa, E., & Giovagnoni, A. (2025) · Clinical & Translational Oncology · doi:10.1007/s12094-025-03939-5 · Cited by 49
“After completing the training and testing phases, there are several metrics that can indicate the quality of the developed model [88].”
Kovalev, M. S., & Utkin, L. V. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.08.007 · Cited by 49
Applying interpretable deep learning models to identify chronic cough patients using EHR data
Luo, X., Gandhi, P., Zhang, Z., Shao, W., Han, Z., Chandrasekaran, V., Turzhitsky, V. M., Bali, V., Roberts, A., Metzger, M., et al. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106395 · Cited by 45
Bajwa, A., Tonoy, A. A. R., Rana, S., & Ahmed, I. (2025) · doi:10.63125/1cr1kj17 · Cited by 47
From Black Box to Glass Box: A Practical Review of Explainable Artificial Intelligence (XAI)
Liu, X., Huang, D., Yao, J., Dong, J., Song, L., Wang, H., Yao, C., & Chu, W. (2025) · AI · doi:10.3390/ai6110285 · Cited by 56
Yu, Y., Huang, J., Liu, S., Zhu, J., & Liang, S. (2021) · Measurement · doi:10.1016/j.measurement.2021.109340 · Cited by 42
Explainable machine learning for sleep apnea prediction
Troncoso-García, A. R., Martínez-Ballesteros, M., Alvarez, F. M., & Troncoso, A. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.351 · Cited by 46
Nai, R., Meo, R., Morina, G., & Pasteris, P. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105887 · Cited by 46
Gebreyesus, Y., Dalton, D., Chiara, D. D., Chinnici, M., & Chinnici, A. (2024) · Electronics · doi:10.3390/electronics13091628 · Cited by 48
Why we do need explainable AI for healthcare
Cinà, G., Röber, T. E., Goedhart, R., & Birbil, Ş. İ. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-025-00209-4 · Cited by 51
On the Soundness of XAI in Prognostics and Health Management (PHM)
Solis‐Martín, D., Galán‐Paez, J., & Borrego‐Díaz, J. (2023) · Information · doi:10.3390/info14050256 · Cited by 48
Counterfactual Learning on Graphs: A Survey
Guo, Z., Wu, Z., Xiao, T., Aggarwal, C. C., Liu, H., & Wang, S. (2025) · Machine Intelligence Research · doi:10.1007/s11633-024-1519-z · Cited by 51
“However, recent works on graph counterfactual learning have shown great potential to overcome the aforementioned challenges on fairness [121], explanation [25] and etc.”
Concept Embedding Analysis: A ReviewPreprint
Schwalbe, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.13909 · Cited by 45
A Comparative Study and Systematic Analysis of XAI Models and their Applications in Healthcare
Gupta, J., & Seeja, K. R. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10103-9 · Cited by 62
“This is crucial when using AI to enhance decision-making in critical fields like healthcare [6, 7].”
InfluentialPanda, M., & Mahanta, S. R. (2024) · CRC Press eBooks · doi:10.1201/9781003442509-6 · Cited by 46
XAI for Churn Prediction in B2B Models: A Use Case in an Enterprise Software Company
Díaz, G. M., Galán-Hernández, J. J., & Carrasco, R. A. (2022) · Mathematics · doi:10.3390/math10203896 · Cited by 41
“Finally, we have agnostic methods, whose implementation does not depend on the ML model used [50].”
BackgroundMethoddel Campo-Ávila, J., Takilalte, A., Bifet, A., & Mora‐López, L. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114147 · Cited by 38
Feature-Weighted Counterfactual-Based Explanation for Bankruptcy Prediction
Cho, S. H., & Shin, K.-S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119390 · Cited by 48
A Survey of Computer Vision Technologies in Urban and Controlled-environment Agriculture
Luo, J., Li, B., & Leung, C. (2023) · ACM Computing Surveys · doi:10.1145/3626186 · Cited by 50
“The lack of human-understandable explanations leads to difficulties in the verification and trust of network decisions [45, 330].”
BackgroundKe, L., Qiu, M., Chen, Z., Zhou, J., Feng, Z., & Long, J. (2023) · Composite Structures · doi:10.1016/j.compstruct.2023.117639 · Cited by 40
Detection of AI-Generated Synthetic Images with a Lightweight CNN
Lađević, A. L., Kramberger, T., Kramberger, R., & Vlahek, D. (2024) · AI · doi:10.3390/ai5030076 · Cited by 46
The natural language explanation algorithms for the lung cancer computer-aided diagnosis system
Meldo, A., Utkin, L., Kovalev, M., & Kasimov, E. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101952 · Cited by 39
Manh, V., Zhou, J., Jia, X., Lin, Z., Xu, W., Mei, Z., Dong, Y., Yang, X., Huang, R., & Ni, D. (2022) · IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control · doi:10.1109/tuffc.2022.3190012 · Cited by 38
Benk, G. Y., Badur, B., & Mardikyan, S. (2022) · Information · doi:10.3390/info13080373 · Cited by 41
An Explainable Artificial Intelligence Approach for Remaining Useful Life Prediction
Youness, G., & Aalah, A. (2023) · Aerospace · doi:10.3390/aerospace10050474 · Cited by 44
Predicting and Mitigating Freshmen Student Attrition: A Local-Explainable Machine Learning Framework
Delen, D., Davazdahemami, B., & Dezfouli, E. R. (2023) · Information Systems Frontiers · doi:10.1007/s10796-023-10397-3 · Cited by 43
“Several approaches proposed in the literature (Carvalho et al., 2019; Gilpin et al., 2018) to interpret ML models and determine variables’ importance essentially provide high-level guidelines for practitioners based on the entire (average) sample characteristics.”
Yavari, A., Mirza, I. B., Bagha, H., Korala, H., Dia, H., Scifleet, P., Sargent, J., Tjung, C., & Shafiei, M. (2023) · Sensors · doi:10.3390/s23187971 · Cited by 43
“In this paper, we have only focused on global interpretability, when explaining CO 2 predictions [47].”
MethodMachine learning applications in SEM-based pore analysis: a review
Papia, E.-M., Kondi, A., & Constantoudis, V. (2025) · Microporous and Mesoporous Materials · doi:10.1016/j.micromeso.2025.113675 · Cited by 44
Fel, T., Vigouroux, D., Cadène, R., & Serre, T. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00163 · Cited by 67
“Several criteria have been proposed to evaluate the quality of explanations [55, 31, 39, 19, 2, 9, 15].”
BackgroundNúñez, J., Cortés, C. B., & Yáñez, M. A. (2023) · Water · doi:10.3390/w15193369 · Cited by 39
Risk-return modelling in the p2p lending market: Trends, gaps, recommendations and future directions
Garzón, M. J. A., Camacho‐Miñano, M., Vargas, M. J. S., & Arroyo, J. (2021) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2021.101079 · Cited by 37
“In addition, Chen et al. (2019) makes a proposal with logistic quantile regression (LQR) for the same objective variable and with explanatory elements.”
MethodExplainability Auditing for Intelligent Systems: A Rationale for Multi-Disciplinary Perspectives
Langer, M., Baum, K. L., Hartmann, K., Hessel, S., Speith, T., & Wahl, J. (2021) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00030 · Cited by 45
“…understanding of system processes and outputs can help to improve system usability for users [9], can aid developers to increase system robustness [10], can help regulating bodies to clarify legal and ethical accountability in case of unfavorable outcomes [11], and can lead to more warranted…”
AutoML for Feature Selection and Model Tuning Applied to Fault Severity Diagnosis in Spur Gearboxes
Cerrada, M., Trujillo, L., Hernández, D. E., Zevallos, H. A. C., Macancela, J.-C., Cabrera, D. R., & Sánchez, R. (2022) · Mathematical and Computational Applications · doi:10.3390/mca27010006 · Cited by 40
“Guaranteeing that learned models are interpretable is one of the main open challenges in ML [75], which can be effected, for example, by model size [76].”
BackgroundHuman Interpretation of Saliency-based Explanation Over Text
Schuff, H., Jacovi, A., Adel, H., Goldberg, Y., & Vu, N. T. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533127 · Cited by 59
“2 FEATURE-ATTRIBUTION EXPLANATIONS Feature-attribution explanations aim to conveywhich parts of the input to a model decision are “important”, “responsible” or “influential” to the decision [3, 7, 36, 42, 60].”
BackgroundZhao, Y., Gao, G., Ding, G., Wang, L., Chen, Y., Zhao, Y., Yu, M., & Zhang, Y. (2022) · CATENA · doi:10.1016/j.catena.2022.106324 · Cited by 38
A Deep Learning-Based Approach for Road Surface Damage Detection
Kulambayev, B. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.029544 · Cited by 36
“In addition, previous research with the usage of deep learning [30–33] only detects the presence or absence of damage.”
Explainable Stacking-Based Model for Predicting Hospital Readmission for Diabetic Patients
Lu, H., & Uddin, S. (2022) · Information · doi:10.3390/info13090436 · Cited by 39
Scientometric Analysis of Artificial Intelligence (AI) for Geohazard Research
Jiang, S., Ma, J., Liu, Z., & Guo, H. (2022) · Sensors · doi:10.3390/s22207814 · Cited by 37
“Based on previous studies, research methods for interpreting AI techniques are maturing [124,125], terminology and metrics are being harmonized [126,127], and there is some development in the evaluation of interpretable AI and interpretation of AI.”
MethodData-driven models for predicting community changes in freshwater ecosystems: A review
Lee, D., Lee, D., Cha, Y., Min, J.-H., & Park, Y. (2023) · Ecological Informatics · doi:10.1016/j.ecoinf.2023.102163 · Cited by 42
The design of reciprocal learning between human and artificial intelligence
Zagalsky, A., Te'eni, D., Yahav, I., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479587 · Cited by 35
“More recently, researchers began exploring explainability and explainable AI (XAI) techniques [8, 11, 20, 51, 60].”
BackgroundA Full Population Auditing Method Based on Machine Learning
Chen, Y., Wu, Z., & Yan, H. (2022) · Sustainability · doi:10.3390/su142417008 · Cited by 39
Toward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 50
Wu, N., Li, Y., Yang, H., Chen, H., Dai, S. X., Hao, C., Yu, C., & Xie, Y. (2024) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3661308 · Cited by 43
“…to identify and expose potential problems during training, encode expert knowledge and intuitions into models, and ensure the fidelity of predictions, interpretable ML [46, 109] is expected to offer more confidence, reliability, and security for the decisions made in simulation-based verification.”
Assessing Decision Tree Stability: A Comprehensive Method for Generating a Stable Decision Tree
Lee, J., Sim, M. K., & Hong, J.-S. (2024) · IEEE Access · doi:10.1109/access.2024.3419228 · Cited by 39
“From an interpretability perspective, the practical usefulness of a model encompasses various aspects including the algorithm’s reliability, transparency, causality, transferability, and fairness [2], [44].”
BackgroundSurvNAM: The machine learning survival model explanation
Utkin, L. V., Satyukov, E. D., & Konstantinov, A. V. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.12.015 · Cited by 48
Nieto-del-Amor, F., Beskhani, R., Ye-Lin, Y., Garcia‐Casado, J., Diaz‐Martinez, A., Ortiz, I. R. M., Diago, V., Hao, D., & Prats-Boluda, G. (2021) · Sensors · doi:10.3390/s21186071 · Cited by 37
“The complex classification algorithms, which can only be interpreted by experts, such as artificial neural network and/or support vector machines, can be dispensed with, which will considerably improve the transferability of the technique to clinical practice [62].”
BackgroundMethodLenatti, M., Moreno-Sánchez, P. A., Polo, E. M., Mollura, M., Barbieri, R., & Paglialonga, A. (2022) · American Journal of Audiology · doi:10.1044/2022_aja-21-00194 · Cited by 39
An evolutionary machine learning algorithm for cardiovascular disease risk prediction
Ordikhani, M., Abadeh, M. S., Prugger, C., Hassannejad, R., Mohammadifard, N., & Sarrafzadegan, N. (2022) · PLoS ONE · doi:10.1371/journal.pone.0271723 · Cited by 40
“On the other hand, white boxes have low accuracy but a clearer interpretation [26, 27].”
BackgroundXie, H., Liu, L., & Yue, H. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192113833 · Cited by 35
“This makes it difficult for users to understand the model’s results [47], thus making it hard to spot errors and improve model performances [48].”
Statistical Data Privacy: A Song of Privacy and Utility
Slavković, A., & Seeman, J. (2022) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033121-112921 · Cited by 48
Gerlach, J., Hoppe, P. E., Jagels, S., Licker, L., & Breitner, M. H. (2022) · Electronic Markets · doi:10.1007/s12525-022-00603-6 · Cited by 39
A nascent design theory for explainable intelligent systems
Herm, L.-V., Steinbach, T., Wanner, J., & Janiesch, C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00606-3 · Cited by 41
“In addition, it is essential to pay attention to reducing the cognitive effort for the user when designing individual EIS components (Carvalho et al. 2019; Cheng et al. 2019).”
BackgroundHow to find a good explanation for clustering?
Bandyapadhyay, S., Fomin, F. V., Golovach, P. A., Lochet, W., Purohit, N., & Simonov, K. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103948 · Cited by 44
FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning
Akli, A., Haben, G., Habchi, S., Papadakis, M., & Traon, Y. L. (2023) · doi:10.1109/ast58925.2023.00018 · Cited by 36
“On top of that, providing developers with information about how the model came to its prediction can enhance the model adoption [47].”
BackgroundIslam, M. M., & Miraz, A. I. (2022) · doi:10.63125/k22shk74 · Cited by 39
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
“As AI development proceeds to the testing phase, incorporating domain experts and users into the testing process ensures a comprehensive assessment against relevant criteria and contexts [1, 43].”
Big Data and Artificial Intelligence: Opportunities and Threats in Electrophysiology
van de Leur, R. R., Boonstra, M., Bagheri, A., Roudijk, R. W., Sammani, A., Taha, K., Doevendans, P. A., van der Harst, P., van Dam, P. M., Hassink, R. J., et al. (2020) · Arrhythmia & Electrophysiology Review · doi:10.15420/aer.2020.26 · Cited by 36
“But interpretability remains important to investigate false positives and negatives, to detect biased or overfitted models, to improve trust in new models or to use the algorithms as a feature detector.(95) Within electrophysiology, few studies have investigated how the AI algorithms came to a certain result.”
MethodNoiseGrad — Enhancing Explanations by Introducing Stochasticity to Model Weights
Bykov, K., Hedström, A., Nakajima, S., & Höhne, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20561 · Cited by 53
Moreno-Sánchez, P. A. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00100 · Cited by 41
Interpretable machine learning to forecast hypoxia in a lagoon
Politikos, D. V., Petasis, G., & Katselis, G. N. (2021) · Ecological Informatics · doi:10.1016/j.ecoinf.2021.101480 · Cited by 35
Review of Applications of Artificial Intelligence Algorithms in Pavement Management
Yang, X., & Zhang, Z. (2022) · Journal of Transportation Engineering Part B Pavements · doi:10.1061/jpeodx.0000373 · Cited by 47
Supervised Machine Learning-Based Decision Support for Signal Validation Classification
Imran, M., Bhatti, A., King, D. M., Lerch, M., Dietrich, J., Doron, G., & Manlik, K. (2022) · Drug Safety · doi:10.1007/s40264-022-01159-2 · Cited by 40
FCE: Feedback Based Counterfactual Explanations for Explainable AI
Suffian, M., Graziani, P., Alonso, J. M., & Bogliolo, A. (2022) · IEEE Access · doi:10.1109/access.2022.3189432 · Cited by 36
DeepFixCX : Explainable privacy‐preserving image compression for medical image analysis
Gaudio, A., Smailagic, A., Faloutsos, C., Mohan, S., Johnson, E., Liu, Y., Costa, P., & Campilho, A. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1495 · Cited by 45
“Good explanations expose bias in order to, for example, allow end-users to trust a prediction (Vellido, 2020), trust a model (Burkart & Huber, 2021; Carvalho et al., 2019; Gilpin et al., 2018), ensure fairness and safety (Chora (cid:1) s et al., 2020), justify decisions (Biran & Cotton, 2017), or…”
BackgroundShen, M. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.05302 · Cited by 43
“trust ML systems [is increasing], for which interpretability is indispensable" [17].”
BackgroundUse of physics to improve solar forecast: Part II, machine learning and model interpretability
Liu, W., Liu, Y., Zhang, T., Han, Y., Zhou, X. S., Xie, Y., & Yoo, S. (2022) · Solar Energy · doi:10.1016/j.solener.2022.08.040 · Cited by 40
eXplainable Artificial Intelligence in Process Engineering: Promises, Facts, and Current Limitations
Bonito, L. P. D., Campanile, L., Natale, F. D., Mastroianni, M., & Iacono, M. (2024) · Applied System Innovation · doi:10.3390/asi7060121 · Cited by 35
Predicting Fluorescence Emission Wavelengths and Quantum Yields via Machine Learning
Souza, R. C., Duarte, J. C., Goldschmidt, R. R., & Borges, I. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02403 · Cited by 33
Barbado, A., & Corcho, Ó. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105222 · Cited by 38
“A relevant research on XAI metrics is [12], where authors propose a taxonomy of properties for different explain-ability scenarios that depend on the use case and the audience of the explanations.”
BackgroundMethodVeldhuis, M. S., Ariëns, S., Ypma, R. J. F., Abeel, T., & Benschop, C. C. G. (2021) · Forensic Science International Genetics · doi:10.1016/j.fsigen.2021.102632 · Cited by 42
Cybersecurity Knowledge Extraction Using XAI
Šarčević, A., Pintar, D., Vranić, M., & Krajna, A. (2022) · Applied Sciences · doi:10.3390/app12178669 · Cited by 41
Data and model bias in artificial intelligence for healthcare applications in New Zealand
Yogarajan, V., Dobbie, G., Leitch, S. V., Keegan, T. T., Bensemann, J., Witbrock, M., Asrani, V. M., & Reith, D. M. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.1070493 · Cited by 36
“On the other hand, it is also important to point out the potential trade off between interpretability, efficacy and equity (Rüping, 2006; Carvalho et al., 2019).”
BackgroundRequirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 38
“However, several research articles have found that the ML community uses the term ‘‘interpretable’’ more often than the term ‘‘explainable’’ [69], [70].”
BackgroundInfluentialTensor Networks for Interpretable and Efficient Quantum-Inspired Machine Learning
Ran, S.-J., & Su, G. (2023) · Intelligent Computing · doi:10.34133/icomputing.0061 · Cited by 40
“However, the dilemma between the interpretability and efficiency, which is a long-concerned topic [1–4], has caused several severe challenges.”
BackgroundElkhawaga, G., Elzeki, O. M., Abu-Elkheir, M., & Reichert, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3357041 · Cited by 37
“According to [6], consistency of explanations corresponds to the similarity between the explanations created for similar predictions, which were made by two different ML models.”
BackgroundResultOgunseyi, T. B., Thiyagarajan, G., He, H., Bist, V., & Du, Z. (2026) · Sensors · doi:10.3390/s26020363 · Cited by 38
“However, this focused approach also limits its applicability to only the specific model for which it was designed [80].”
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.