Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 801–900 of 1,007
A Hierarchical Approach to Interpretability of TS Rule-Based Models
Witold Pedrycz, C., Gacek, A., & Wang, X. (2021) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2021.3094661 · Cited by 11
Music, Art, Machine Learning, and Standardization
Brook, T. (2021) · Leonardo · doi:10.1162/leon_a_02135 · Cited by 13
Cali, Ü., Kuzlu, M., Pipattanasomporn, M., Kempf, J., & Bai, L. (2021) · Springer eBooks · doi:10.1007/978-3-030-83301-5_6 · Cited by 11
Explainable Artificial Intelligence for Authorship Attribution on Social Media
Theóphilo, A., Padilha, R., Andaló, F., & Rocha, A. D. R. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746262 · Cited by 11
Predicting chatter using machine learning and acoustic signals from low-cost microphones
John, S. S., Alberts, M., Karandikar, J. M., Coble, J. B., Jared, B. H., Schmitz, T. L., Ramsauer, C. M., Leitner, D., & Khojandi, A. (2023) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-023-10918-z · Cited by 13
“Interpretability of machine learning algorithms is a topic receiving significant focus [27].”
BackgroundHow to Open a Black Box Classifier for Tabular Data
Walters, B., Ortega‐Martorell, S., Olier, I., & Lisboa, P. J. G. (2023) · Algorithms · doi:10.3390/a16040181 · Cited by 11
“Interpretability This can be referred to a formal framework involving the three Cs of interpretability [23]: • Completeness—the proposed models have global coverage in the sense that they provide a direct and causal explanation of the model output from the input data, over the complete range of input data.”
BackgroundMethodInfluentialFair pairwise learning to rank
Cerrato, M., Köppel, M., Segner, A., Esposito, R., & Krämer, S. (2020) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa49011.2020.00083 · Cited by 17
“Model explanation, despite being a relevant and very active line of research [1], [7], is far from being a solved problem, especially for large complex models with millions of parameters.”
BackgroundMachine learning explanability method for the multi-label classification model
Singla, K., & Biswas, S. (2021) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc50631.2021.00063 · Cited by 11
“[3] Shows that as the machine learning based models become an integral part of the human life.”
BackgroundSystematic Literature Review: XAI and Clinical Decision Support
Connolly, T. M., Soflano, M., & Papadopoulos, P. (2022) · Advances in medical technologies and clinical practice book series · doi:10.4018/978-1-6684-5092-5.ch008 · Cited by 14
A Literature Review and Research Agenda on Explainable Artificial Intelligence (XAI)
Kalyanathaya, K. P., & K., K. P. (2022) · International Journal of Applied Engineering and Management Letters · doi:10.47992/ijaeml.2581.7000.0119 · Cited by 10
Widmer, C. L., Sarker, M. K., Nadella, S., Fiechter, J., Juvina, I., Minnery, B., Hitzler, P., Schwartz, J., & Raymer, M. (2023) · Journal of Web Semantics · doi:10.1016/j.websem.2023.100807 · Cited by 10
New optimization models for optimal classification trees
Alès, Z., Huré, V., & Lambert, A. (2023) · Computers & Operations Research · doi:10.1016/j.cor.2023.106515 · Cited by 18
“1 Introduction”
Random Forest Variable Importance Measures for Spatial Dynamics: Case Studies from Urban Demography
Georgati, M., Hansen, H. S., & Keßler, C. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12110460 · Cited by 10
Ramokapane, K. M., & Rashid, A. (2023) · New Security Paradigms Workshop · doi:10.1145/3633500.3633503 · Cited by 11
“Previous research [9, 11, 12, 23, 33, 44] has proposed several desirable properties of satisfactory explanations.”
BackgroundZharova, A., Boer, A., Knoblauch, J., Schewina, K., & Vihs, J. (2024) · Environmental Data Science · doi:10.1017/eds.2024.8 · Cited by 10
Zhang, Q., Yang, R., Wang, Z., Li, Y., Dong, F., Liu, J., Xiong, S., Wang, G., & Feng, J. (2025) · Energy storage materials · doi:10.1016/j.ensm.2025.104261 · Cited by 13
Synthesizing Explainability Across Multiple ML Models for Structured Data
Veledar, E., Zhou, L., Veledar, O., Gardener, H. E., Gutierrez, C. M., Romano, J. G., & Rundek, T. (2025) · Algorithms · doi:10.3390/a18060368 · Cited by 11
“Furthermore, new regulations and the requirements of highly regulated fields mandate the suitability and verifiability of algorithmic decisions [3].”
Ahmed, M. J., Mozó, A., & Karamchandani, A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3097 · Cited by 15
XAI-Eval: A framework for comparative evaluation of explanation methods in healthcare
Agrawal, K., Shawi, R. E., & Ahmed, N. M. O. S. (2025) · Digital Health · doi:10.1177/20552076251368045 · Cited by 11
Yao, B., Yu, X., Zhang, Y., Chen, J., Zhu, X., Zhang, C., & Jijun, T. (2025) · Frontiers in Endocrinology · doi:10.3389/fendo.2025.1719631 · Cited by 11
"Is It My Turn?": Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
Benk, M., Weibel, R. P., Feuerriegel, S., & Ferrario, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555580 · Cited by 14
“ML interpretability [16, 49, 54, 56, 65] refers to the scientific efforts aiming at the design of methods to validate ML models, explain their inner workings and their predictions by means, for example, of the exploration of outcome visualizations or the use of counterfactual explanations [72].”
MethodPinto, N., Amaral, J. L. M., Lopes, A. J., & de Melo, P. L. (2023) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-023-00777-0 · Cited by 10
“This characteristic of expressing the behavior of a system com-prehensibly is called interpretability and does not have a performance metric to evaluate [9].”
BackgroundFine-Tuning Languages: Epistemological Foundations for Ethical AI in Journalism
Dierickx, L., & Lindén, C. (2023) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds57534.2023.00013 · Cited by 12
“This metric, used for classification tasks [42], is not the only tool to evaluate a model performance: the kappa statistic indicates the agreement between the model’s prediction and the true values, the true positive rate (or sensitivity) measures the proportion of correct predictions, the true…”
BackgroundMethodInfluentialKega, I., Nderu, L., Mwangi, R. W., & Njagi, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121092 · Cited by 12
Xiong, M., Wang, H., Che, C., & Lin, R. (2023) · Proceedings of the Institution of Mechanical Engineers Part O Journal of Risk and Reliability · doi:10.1177/1748006x231205498 · Cited by 10
“At the same time, these methods ignore the model explainability 17,18 and are generally ‘‘black box’’ models, making them difficult to be applied in practical engineering.”
Branco, R., Agostinho, C., Gusmeroli, S., Lavasa, E., Dikopoulou, Z., Monzó, D., & Lampathaki, F. (2023) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc58018.2023.10332373 · Cited by 12
“Explainable post-hoc AI pipeline, from [18] The main advantage of model-agnostic interpretation methods is their flexibility since they can be applied to any machine learning model [19].”
MethodHong, J., Park, K. H., & Pavlic, T. P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00481 · Cited by 19
“In particular, the debate over what degree of interpretability can be ascribed to attention weights over input tokens still needs to be settled to help meet the need for interpretability of attention mechanisms [1,9,40,71,86].”
BackgroundKakati, D., & Banerjee, R. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115516 · Cited by 12
Holistic Explainability Requirements for End-to-End Machine Learning in IoT Cloud Systems
Nguyen, M.-L., Phung, T., Ly, D.-H., & Truong, H. (2021) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00034 · Cited by 11
Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics
Ghorbani, A., Berenbaum, D., Ivgi, M., Dafna, Y., & Zou, J. (2021) · Information · doi:10.3390/info13010015 · Cited by 15
Yokoyama, S., Kagawa, F., Takamura, M., Takagaki, K., Kambara, K., Mitsuyama, Y., Shimizu, A., Okada, G., & Okamoto, Y. (2023) · BMC Public Health · doi:10.1186/s12889-023-14984-6 · Cited by 12
“While this does not interfere with the motivation for predictionoriented machine learning models, it does make it difficult to understand the rationale behind the predictions [13].”
BackgroundAssessing TD Macro-Management: A Nested Modeling Statistical Approach
Nikolaidis, N., Mittas, N., Ampatzoglou, A., Arvanitou, E.-M., & Chatzigeorgiou, A. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3237460 · Cited by 10
“The reason for this is that Machine Learning models suffer from low interpretability [7], whereas the goal of the study is to explore the TD Micro-and Macro-Management parameters and identify the most prevalent ones.”
BackgroundFairness, explainability, privacy, and robustness for trustworthy algorithmic decision-making
Majumdar, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85713-0.00017-7 · Cited by 10
Evaluation of Machine Learning on Smart Home Data for Prediction of Electrical Energy Consumption
Widianto, M. H., Gunawan, A. A. S., Heryadi, Y., & Budiharto, W. (2023) · International Conference on Computer Science, Information Technology and Engineering (ICCoSITE) · doi:10.1109/iccosite57641.2023.10127700 · Cited by 10
Alarcon, G. M., & Willis, S. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.134 · Cited by 18
Machine learning for polyphenol-based materials
Jiang, S., Yang, P., Zheng, Y., Lu, X., & Xie, C. (2024) · Smart Materials in Medicine · doi:10.1016/j.smaim.2024.02.001 · Cited by 14
Machine Learning Interpretability Techniques for Outage Prediction: A Comparative Study
Jalali, A., Schindler, A., Haslhofer, B., & Rauber, A. (2020) · PHM Society European Conference · doi:10.36001/phme.2020.v5i1.1244 · Cited by 12
Transparent AI: explainability of deep learning based load disaggregation
Murray, D., Stanković, L., & Stanković, V. (2021) · ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys) · doi:10.1145/3486611.3492410 · Cited by 13
“ACM ISBN 978-1-4503-9114-6/21/11... $15.00 https://doi.org/10.1145/3486611.3492410 feature articles [1, 3, 5, 6, 8, 11], most explainable AI implementations focus on technology designs, e.g., for the purpose of removing the bugs in the code or improving the models, while other potential users of…”
BackgroundInfluentialHuman in the Loop Fuzzy Pattern Tree Evolution
Murphy, A., Murphy, G., Dias, D. M., Amaral, J. L. M., Naredo, E., & Ryan, C. (2022) · SN Computer Science · doi:10.1007/s42979-022-01044-w · Cited by 10
“This must come with no, or minimal, impact on the models performance [4].”
BackgroundKRAMER: Interpretable Rarity Meter for Crypto Collectibles
Krasnoselskii, M., Madhwal, Y., & Yanovich, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3236080 · Cited by 10
“A score function is a rarity meter but also expert-based and interpretable [22], [23].”
BackgroundXAI Method Properties: A (Meta-)study.Preprint
Schwalbe, G., & Finzel, B. (2021) · arXiv (Cornell University) · Cited by 10
Li, J., Fang, L., Liu, Y., Xie, J., & Wang, X. (2024) · Behavioral Sciences · doi:10.3390/bs14060477 · Cited by 9
“On one hand, ITS is highly practical, with data models focusing solely on predictive accuracy, while neglecting interpretability [6].”
MethodA Data-Driven Approach to Predicting Recreational Activity Participation Using Machine Learning
Lee, S., & Kang, M. (2024) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2024.2343815 · Cited by 10
“However, machine learning models are black boxes with hidden inner workings, making it difficult for users to understand the relationship between data and results (Carvalho et al., 2019).”
The Reality of High Performing Deep Learning Models: A Case Study on Document Image Classification
Saifullah, S., Agne, S., Dengel, A. R., & Ahmed, S. (2024) · IEEE Access · doi:10.1109/access.2024.3425910 · Cited by 9
“However, despite their extraordinary predictive power, the black-box nature [6], [7] of deep networks remains a major hindrance to their safe integration in industry.”
BackgroundEnhancing Bank Term Deposit Predictions: A Machine Learning Approach with CatBoost and SHAP
Yu, Q. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.19485 · Cited by 10
Utilizing Machine Learning for Early Intervention and Risk Management in the Opioid Overdose Crisis
Tai, A. M. Y., Kazemi, A., Kim, J. J., Schmeckenbecher, J., Kitchin, V., Suen, J., Moro, R., & Krausz, M. R. (2025) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.70008 · Cited by 9
“A recent literature review highlights the complexity of interpreting and selecting the most appropriate metrics to assess the quality of a ML model (Carvalho, Pereira, and Cardoso 2019).”
Qu, X., Jiang, C., Shan, M., Ke, W., Chen, J., Zhao, Q., Hu, Y., Liu, J., Qin, L., & Cheng, G. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01732 · Cited by 9
Interpretable deep learning for deconvolutional analysis of neural signals
Tolooshams, B., Matias, S., Wu, H., Temereanca, S., Uchida, N., Murthy, V. N., Masset, P., & Ba, D. E. (2024) · Neuron · doi:10.1016/j.neuron.2025.02.006 · Cited by 12
“42,43 Our method complements these tools by extracting interpretable impulse-like responses from multiplexed signals in single neurons, recovering locally low-rank structures to characterize heterogeneity and homogeneity across neural populations.”
Floreale, G., Baraldi, P., Zio, E., & Fink, O. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110518 · Cited by 11
“Significant progresses have been made in recent years in the field of eXplainable Artificial Intelligence (XAI), where several techniques have been proposed to explain DL models and understand how their outputs are obtained [8].”
Sheng, Y., Arbabi, H., Ward, W. O. C., & Mayfield, M. (2025) · Energy and Buildings · doi:10.1016/j.enbuild.2025.115723 · Cited by 9
Exploring the Impact of Explainability in Large Language Model (LLM) Applications on User Experience
Wang, Y., Fang, X., Xu, Z., Li, J., & Wang, L. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719941 · Cited by 9
Aeroengine Performance Prediction Using a Physical-Embedded Data-Driven Method
Mo, T., Dai, S., Fu, A., Zhu, X., & Li, S. (2025) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2025.3574667 · Cited by 9
“In our case, this manifests as the presence of specific patterns in the feature summary [52].”
Salmanpour, M. R., Amiri, S., Gharibi, S., Shariftabrizi, A., Xu, Y., Weeks, W. B., Rahmim, A., & Hacihaliloglu, I. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01585-5 · Cited by 14
“This section examines the importance of these algorithms and the methods developed to enhance interpretability, along with the challenges in assessing explanation quality [83–86].”
Sparse-enhanced additive interaction neural network for interpretable credit decision
Lan, X., Fan, H., Liu, W., Xia, M., & Guo, K. (2025) · Decision Support Systems · doi:10.1016/j.dss.2025.114507 · Cited by 9
A Survey on Interpretability in Visual Recognition
Wan, Q., Gao, C., Wang, R., & Chen, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3672629 · Cited by 11
“Perturbation-based methods like Explaining prototypes [64] and CaCE [65] focus on understanding the model’s behavior based on perturbations of input data.”
Tabak, B. M., Cardoso, D. H. R., & Silva, C. C. (2025) · Sustainability · doi:10.3390/su17209219 · Cited by 10
Díaz, G. M. (2025) · AI · doi:10.3390/ai7010003 · Cited by 9
Multimodal artificial intelligence for enhanced skin cancer diagnosis and prognosis
Karnwal, A., Selvaraj, M., Kumar, G., Kumar, A., Al-Tawaha, A. R., Aqueel-Ur-Rehman, & Nesterova, N. (2026) · Discover Oncology · doi:10.1007/s12672-026-04532-0 · Cited by 10
“Several reviews have explored methods to improve explainability, including surveys by [10, 17, 101] categorized these approaches into four types: text-based explanations, visualizations, local explanations, and example-based explanations.”
Gonçalves, A., & Correia, A. C. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6020043 · Cited by 11
Despotović, M., Koch, D., Stumpe, E., Brunauer, W., & Zeppelzauer, M. (2023) · Journal of European real estate research · doi:10.1108/jerer-11-2022-0036 · Cited by 9
Demystifying the black box: an overview of explainability methods in machine learning
Kinger, S., & Kulkarni, V. Y. (2023) · International Journal of Computers and Applications · doi:10.1080/1206212x.2023.2285533 · Cited by 12
Using Decision Trees for Interpretable Supervised Clustering
Kokash, N., & Махнист, Л. П. (2024) · SN Computer Science · doi:10.1007/s42979-023-02590-7 · Cited by 10
“With the adoption of Machine Learning (ML) methods for decision-making in industries with high accountability demands such as healthcare or financial services, we observed a spurge in interest for model interpretability [6, 28].”
Subgroup Discovery with Small and Alternative Feature Sets
Bach, J. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725358 · Cited by 10
Deep learning inspired game-based cognitive assessment for early dementia detection
Maji, P. K., Acharya, S., Paul, P., Chakraborty, S., & Basu, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109901 · Cited by 13
How robust are ensemble machine learning explanations?
Calzarossa, M. C., Giudici, P., & Zieni, R. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129686 · Cited by 10
An Enhancement of Object Detection Using YOLO V8 and Mobile Net in Challenging Conditions
Pasupuleti, S., Ramalakshmi, K., Gunasekaran, H., Arokiaraj, A. R. M., Debnath, S., & Jebaseeli, T. J. (2025) · SN Computer Science · doi:10.1007/s42979-025-03856-y · Cited by 9
“Romero et al., [15] state that because of its simplified architecture, which can be effectively deployed on devices with limited resources without compromising performance, it is a well-liked option for mobile and edge computing applications.”
A Systematic Review of User-Centred Evaluation of Explainable AI in HealthcarePreprint
Donoso-Guzmán, I., Kacafírková, K. S., Szymanski, M., Jacobs, A., Parra, D., & Verbert, K. (2025) · VUBIR (Vrije Universiteit Brussel) · doi:10.48550/arxiv.2506.13904 · Cited by 12
A Review of Explainable Artificial Intelligence
Lin, K., Liu, Y., Li, L., & Dou, R. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-85910-7_61 · Cited by 10
Mei, F., Li, X., Zheng, J., Sha, H., & Li, D. (2022) · IET Generation Transmission & Distribution · doi:10.1049/gtd2.12385 · Cited by 9
Explaining Preference-Driven Schedules: The EXPRES Framework
Pozanco, A., Mosca, F., Zehtabi, P., Magazzeni, D., & Kraus, S. (2022) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v32i1.19861 · Cited by 9
Sensitivity analysis of factors controlling earth fissures due to excessive groundwater pumping
Li, Y., Friedman, N., Teatini, P., Benczúr, A. A., Ye, S., Zhu, L., & Zoccarato, C. (2022) · Stochastic Environmental Research and Risk Assessment · doi:10.1007/s00477-022-02237-8 · Cited by 10
“…such as Shapley Additive Explanations and Mean Decrease Accuracy (MDA) were designed to assess feature importance according to their relevance for the corresponding estimator, similarly to the key insights of GSA (Arabameri et al 2022; Breiman 2001; Carvalho et al 2019; Lundberg and Lee 2017).”
MethodOn the Need for Collaborative Intelligence in Cybersecurity
Martin, T. (2022) · Electronics · doi:10.3390/electronics11132067 · Cited by 11
“There are a number of good surveys and overviews in explainable AI, including [18]; [19]; [20]).”
BackgroundDisentangling error structures of precipitation datasets using decision trees
Sui, X., Li, Z., Tang, G., Yang, Z., & Niyogi, D. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113185 · Cited by 9
Explaining Deep Learning Models for Per-packet Encrypted Network Traffic Classification
Garcia, L., Bartlett, G., Ravi, S., Ibrahim, H., Hardaker, W., & Kline, E. (2022) · IEEE International Symposium on Measurements & Networking (M&N) · doi:10.1109/mn55117.2022.9887744 · Cited by 12
“For models that are not inherently interpretable, understanding deep learning models involves post-hoc explanations where the attributions of a given sample’s features relative to its output classification are overlaid on a sample visualization [5].”
BackgroundReCCoVER: Detecting Causal Confusion for Explainable Reinforcement Learning
Gajcin, J., & Dusparić, I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15565-9_3 · Cited by 9
Methods and Metrics for Explaining Artificial Intelligence Models: A Review
Banerjee, P., & Barnwal, R. P. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-12807-3_4 · Cited by 14
Balbino, M. D. S., Zárate, L. E., & Nobre, C. N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120373 · Cited by 10
From Discovery to Adoption: Understanding the ML Practitioners’ Interpretability Journey
Ashtari, N., Mullins, R. S., Qian, C., Wexler, J., Tenney, I., & Pushkarna, M. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596046 · Cited by 13
“Enhancing model performance is one of the key objectives, as emphasized in prior work [16].”
BackgroundDealing with Explainability Requirements for Machine Learning Systems
Li, T., & Han, L. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00182 · Cited by 12
“The further division of explainability methods differs slightly [13], [5].”
BackgroundMethodSupporting Safety Analysis of Image-processing DNNs through Clustering-based Approaches
Attaoui, M. O., Fahmy, H. M., Pastore, F., & Briand, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643671 · Cited by 12
“, coverage above 40% for the range [0-5], and above 70% for the range [5-10]) and leads to 3.”
BackgroundInterpretability and Explainability of Machine Learning Models: Achievements and Challenges
Henriques, J. M. O., Rocha, T., Carvalho, P., Silva, C., & Paredes, S. (2024) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-59216-4_9 · Cited by 13
Loftus, J. R., Bynum, L., & Hansen, S. (2024) · neural information processing systems · doi:10.52202/079017-3579 · Cited by 18
“The idea is general, but we are motivated by explaining or interpreting AI or machine learning models [6, 13, 14, 27].”
MethodInterpretable Biometrics: Should We Rethink How Presentation Attack Detection is Evaluated?
Sequeira, A. F., Silva, W., Pinto, J. R., Gonçalves, T., & Cardoso, J. S. (2020) · International Workshop on Biometrics and Forensics (IWBF) · doi:10.1109/iwbf49977.2020.9107949 · Cited by 9
“Thus, further efforts should be devoted to reducing subjectivity in the evaluation of explanations, through objective metrics and procedures [30].”
BackgroundSelf-citationBurst Imaging for Light-Constrained Structure-From-Motion
Ravendran, A., Bryson, M., & Dansereau, D. G. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3137520 · Cited by 9
Finding Representative Interpretations on Convolutional Neural Networks
Lam, P. C.-H., Chu, L., Torgonskiy, M., Pei, J., Zhang, Y., & Wang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00138 · Cited by 10
“We focus on evaluating the representativeness of interpretations, that is, how well an interpretation for an input image can be reused to interpret the predictions on new images that are similar to the input image [7].”
BackgroundMethodInfluentialSchwenke, L., & Atzmueller, M. (2021) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa53316.2021.9564126 · Cited by 12
Transparent single-cell set classification with kernel mean embeddings
Shan, S., Baskaran, V. A., Yi, H., Ranek, J. S., Stanley, N., & Oliva, J. B. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545538 · Cited by 10
“An important advantage of interpretable models is their capacity to uncover scientific knowledge [7].”
BackgroundRepresentativity and Consistency Measures for Deep Neural Network ExplanationsPreprint
Fel, T., & Vigouroux, D. (2020) · arXiv (Cornell University) · Cited by 15
Explaining ECG Biometrics: Is It All In The QRS?
Pinto, J. R., & Cardoso, J. S. (2020) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 9
Self-citationBarbado, A., & Corcho, Ó. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06031 · Cited by 9
Standardized Evaluation of Machine Learning Methods for Evolving Data StreamsPreprint
Haug, J., Tramountani, E., & Kasneci, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.13625 · Cited by 10
“, the General Data Protection Regulation of the European Union), the transparency and interpretability of these models has gained considerable attention [28, 29, 30, 31, 32].”
BackgroundLee, S. (2022) · PDXScholar (Portland State University) · doi:10.15760/etd.7865 · Cited by 13
Why Don't You Do Something About It? Outlining Connections between AI Explanations and User ActionsPreprint
Mansi, G., & Riedl, M. O. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06297 · Cited by 11
Bertrand, A., Eagan, J., Maxwell, W., & Brand, J. L. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642326 · Cited by 10
“Specifcally, “legal explanations”, i.e. explanations designed to support the legal compliance process, have been examined by XAI researchers [4, 15, 25].”
BackgroundPrada, E. J. A., Ojeda, A. V. V., & Ojeda, M. Y. V. (2025) · Cogent Engineering · doi:10.1080/23311916.2025.2467153 · Cited by 8
“In medical contexts, interpretability may enable physicians to make sure that the model makes the right or wrong predictions for the right reasons, so researchers must provide accurate and reliable but also interpretable machine learning models (Carvalho et al., 2019).”
Sepulveda, E. E., Vandervorst, F., Baesens, B., & Verdonck, T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126922 · Cited by 11
David, R. T., Shankar, H., Kura, P., Kowtarapu, K., Maheswari, S. U., & Karkuzhali, S. (2025) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci63694.2025.10988079 · Cited by 21
“Explainable RL frameworks are crucial for autonomous system confidence, especially in safety-critical applications [14,15].”
Surrogate-guided optimization in quantum networks
Prielinger, L., Iñesta, Á. G., & Vardoyan, G. (2025) · npj Quantum Information · doi:10.1038/s41534-025-01048-3 · Cited by 9
“Additionally,theoutputsfromthesecomplex modelsareoftenmoredif fi cult tointerpretcompared tothose from simpler models 29 .”
Sezer, E., Dokuzparmak, E., Özçelik, H., Yaşar, E., Kaya, T., Güner, T., & Akgöl, S. (2025) · ACS Omega · doi:10.1021/acsomega.5c02363 · Cited by 9
Adaptive Explainable AI: Personalizing Machine Explanations Based on User Expertise Levels
Mohammed, A. A. A. (2025) · Journal of Posthumanism · doi:10.63332/joph.v5i7.2793 · Cited by 9
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.