Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 901–1,000 of 1,007
Explainable Artificial Intelligence for Ovarian Cancer: Biomarker Contributions in Ensemble Models
UCUZAL, H., & Kıvrak, M. (2025) · Biology · doi:10.3390/biology14111487 · Cited by 9
“…intelligence (XAI) techniques, notably SHAP and LIME, provide global and patient-level attributions that elucidate how features influence predictions, enabling model debugging, identification of spurious associations, and communication of individualized risk factors to clinicians and patients [6].”
Chen, C., Tan, Y., Liu, Y., Sun, M., Wang, Z. L., Zheng, X., Shen, Z., & Zhao, M. (2026) · Environmental Research · doi:10.1016/j.envres.2026.123853 · Cited by 11
Samra, R. M. A., & Alí, R. R. (2026) · AgriEngineering · doi:10.3390/agriengineering8070270 · Cited by 11
“XAI refers to methods and frameworks that make AI decisions understandable to humans [44].”
Intelligent Machine Learning System for Predicting Customer Churn
He, C., Ding, C. H. Q., Chen, S.-B., & Luo, B. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00085 · Cited by 9
Predicting inmates misconduct using the SHAP approach
de Oliveira, F. M., Balbino, M. D. S., Zárate, L. E., Ngo, F. T., Govindu, R., Agarwal, A., & Nobre, C. N. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09352-z · Cited by 10
The Use of ICT as a Resource and Media for Modern 21st Century Learning in Primary Schools
Prihantini, P., Sutarto, S., Punggeti, R. N., Rukmini, A., & Intes, A. (2023) · Journal International of Lingua and Technology · doi:10.55849/jiltech.v3i1.509 · Cited by 8
“One of the benefits of using Information and Communication Technology (ICT) is that learning becomes more innovative and creative Instructors must then create interesting technology-based learning materials (Carvalho, 2019; Du et al., 2021; Md Enzai et al., 2021).”
BackgroundCAT: Interpretable Concept-based Taylor Additive Models
Duong, V., Wu, Q., Zhou, Z., Zhao, H., Luo, C., Zavesky, E., Yao, H., & Shao, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672020 · Cited by 12
“Earlier studies primarily focused on perturbation-based post-hoc approaches [7, 21, 35], but these methods are either computationally expensive or hard to faithfully represent the model’s behavior [15, 40, 44].”
Data-driven constitutive meta-modeling of nonlinear rheology via multifidelity neural networks
Saadat, M., Hartt, W. H., Wagner, N. J., & Jamali, S. (2024) · Journal of Rheology · doi:10.1122/8.0000831 · Cited by 9
A Surrogate Approach to Explainable AI for Predictive Maintenance: Techniques and Applications
Rakesh, P., Shruti, J. R., Thippeswamy, I., Nithya, B., & Dheeraj, V. (2024) · International Conference on Circuits, Control, Communication and Computing (I4C) · doi:10.1109/i4c62240.2024.10748524 · Cited by 10
Ensemble machine learning model for exploration and targeting of Pb-Zn deposits in Algeria
Remidi, S., Boutaleb, A., Tachi, S. E., Hasnaoui, Y., Szczepanek, R., & Seffari, A. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-01718-y · Cited by 8
“The classification metrics evaluate the model performance of machine learning models by classifying input data point into one or more pre-established categories (Carvalho et al. 2019).”
Explainable Artificial Intelligence (XAI) in Insurance
Verma, N., Varshney, A. K., Singhal, R. K., Gaur, M. P., Garg, A., & Das, S. (2025) · International Conference on Pervasive Computational Technologies (ICPCT) · doi:10.1109/icpct64145.2025.10939062 · Cited by 11
Less Is More: The Influence of Pruning on the Explainability of CNNs
Merkle, F., Weber, D., Schöttle, P., Schlögl, S., & Nocker, M. (2025) · IEEE Access · doi:10.1109/access.2025.3569575 · Cited by 11
“However, Carvalho et al. [18] argue that results from a functionally-grounded approach have low validity, as human feedback is missing and the defined proxies may not fully measure explainability.”
Vision paper: causal inference for interpretable and robust machine learning in mobility analysis
Xin, Y., Tagasovska, N., Pérez‐Cruz, F., & Raubal, M. (2022) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3557915.3561473 · Cited by 9
“Research on interpretable and robust machine learning has gained increasing attention in the last few years, proliferating multiplemethodologies [7, 22].”
BackgroundA personalized classification model using similarity learning via supervised autoencoder
Jo, H., & Jun, C. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109773 · Cited by 8
Over-relaxed multi-block ADMM algorithms for doubly regularized support vector machines
Dai, Y., Zhang, Y., & Wu, Q. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.01.082 · Cited by 8
Patterson, A., Elbasir, A., Tian, B., & Auslander, N. (2023) · Cancers · doi:10.3390/cancers15071958 · Cited by 9
Package Proposal for Data Pre-Processing for Machine Learning Applied to Precision Irrigation
dos Santos, R. P., Beko, M., & Leithardt, V. R. Q. (2023) · Conference on Cloud and Internet of Things (CIoT) · doi:10.1109/ciot57267.2023.10084899 · Cited by 8
“Creating a specific computer program and making it do something similar to what is done with an ML algorithm would demand a considerable amount of work, given the need to program each of the details of the process [5].”
BackgroundDemocratizing Quality-Based Machine Learning Development through Extended Feature Models
d’Aloisio, G., Marco, A. D., & Stilo, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30826-0_5 · Cited by 11
The Societal Impacts of Algorithmic Decision-Making
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195 · Cited by 11
Path-Sampling and Machine Learning for Rare Abnormal Events: Application to Polymerization CSTRs
Sudarshan, V., Seider, W. D., Patel, A. J., Oktem, U. G., & Arbogast, J. E. (2024) · Chemical Engineering Science · doi:10.1016/j.ces.2024.120513 · Cited by 8
Scholz, M., & Bürkner, P. (2025) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2024.2449534 · Cited by 10
“Yarkoni and Westfall [24] and Breiman [25] are examples for the call to put more focus on predictions, even if explanation is the main goal.”
Deep Learning: Systems and Responsibility
Wasay, A., Chatterjee, S., & Idreos, S. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457541 · Cited by 14
“emerged as a sub-field of deep learning that seeks to augment the design, training, and deployment of deep learning models to make them understandable to humans [7, 75, 84].”
Interpretability of Machine Learning-Based Results of Malware Detection Using a Set of Rules
Dolejš, J., & Jureček, M. (2022) · Advances in information security · doi:10.1007/978-3-030-97087-1_5 · Cited by 8
Human In Command Machine Learning
Holmberg, L. (2021) · doi:10.24834/isbn.9789178771875 · Cited by 10
PyTorch, Explain! A Python library for Logic Explained NetworksPreprint
Barbiero, P., Ciravegna, G., Georgiev, D., & Giannini, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11697 · Cited by 9
“…Intelligence (XAI) has become strategic and has been massively encouraged, leading to the development of a variety of techniques that aim at explaining black-box models (Das and Rad, 2020; Brundage et al., 2020) or at developing effective interpretable models (Carvalho et al., 2019; Rudin, 2019).”
BackgroundMoreno-Sánchez, P. A. (2022) · Research Square · doi:10.21203/rs.3.rs-1628347/v1 · Cited by 8
Assessment of the Reliablity of a Model's Decision by Generalizing Attribution to the Wavelet DomainPreprint
Kasmi, G., Dubus, L., Saint‐Drenan, Y., & Blanc, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14979 · Cited by 8
“Attribution methods [5], which consist in identifying the most important features in the input, have improved the understanding of the decision process of deep learning models.”
MethodM2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 91
“Besides performance metrics, the system should provide global explanations on how the model generally works, especially when working with huge datasets [3, 10, 26, 39].”
BackgroundMethodFairUDT: Fairness-aware Uplift Decision Trees
Zahid, A., Ali, A. R., Raza, S., Shahnawaz, R., Kamiran, F., & Karim, A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113068 · Cited by 7
Lawson, C. G. R., Englezos, E., & Rourke, M. F. (2025) · Plants People Planet · doi:10.1002/ppp3.70007 · Cited by 9
“…a level of explainability through constraints imposed on the complexity of the model (called intrinsic explainabil-ity), or procedures and techniques (or methods) are applied after the models are trained to explain the outputs (called post hoc explainabil-ity) (Carvalho et al., 2019, p. 12).”
Bulut, C., & Arslan, E. (2025) · Applied Sciences · doi:10.3390/app15105723 · Cited by 9
Explainable AI for Daily Scenarios from End-Users’ Perspective: Non-Use, Concerns, and Ideal Design
Wang, L., Anyi, C. L., Xu, K., Liu, Y., Arriaga, R. I., & Goel, A. K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735796 · Cited by 9
“XAI design is inherently multifaceted [23, 77, 129], and its effectiveness depends not only on the design of the artifacts themselves but also on contextual factors such as the application scenarios, the characteristics of targeted users, and their situated motivations [64, 77, 117].”
Classifying XAI Methods to Resolve Conceptual Ambiguity
Dib, L., & Capus, L. (2025) · Technologies · doi:10.3390/technologies13090390 · Cited by 9
A multi-AI approach to predicting municipal solid waste generation and recycling demand in Hong Kong
Xu, P. J., & Zheng, H. R. (2025) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2025.108590 · Cited by 7
Towards Human-Centered Early Prediction Models for Academic Performance in Real-World Contexts
Zhang, H., Ren, Y., Nurius, P., Mankoff, J. C., & Dey, A. (2025) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3757433 · Cited by 4
“…is still no universal consensus on the exact definition of explainability and related terms such as interpretability [52, 131], explainability, or Explainable AI (XAI), it generally refers to the ability of a system to make its decisions or behaviors understandable to humans [10, 27, 54, 112].”
Towards precision medicine for otology and neurotology: Machine learning applications and challenges
Adcock, K. S., Arulchelvan, E., Shields, N., & Vanneste, S. (2025) · Hearing Research · doi:10.1016/j.heares.2025.109473 · Cited by 7
From data to diagnosis: a systematic review on AI-driven approaches to diabetes prediction
Masood, S., Albashrawi, M. A., Khan, M. A., & Dwivedi, Y. K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11485-3 · Cited by 13
Owotogbe, J., Oyekanmi, E., Adepoju, S. E., & Akinsunmade, A. E. (2026) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2026.101756 · Cited by 8
Qureshi, M. A., Noor, A. A., Manzoor, A., Mazhar Qureshi, M. D., Younus, A., & Rashwan, W. (2026) · PLOS One · doi:10.1371/journal.pone.0351473 · Cited by 7
Frugal day-ahead forecasting of multiple local electricity loads by aggregating adaptive models
Lambert, G., Hamrouche, B., & de Vilmarest, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-42488-1 · Cited by 7
Machine learning-based electricity theft detection using support vector machines
Abro, S. A., Guanghua, L., Laghari, J. A., Bhayo, M. A., & Memon, A. A. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i2.pp1240-1250 · Cited by 7
Explainable Multistage Ensemble 1D Convolutional Neural Network for Trust Worthy Credit Decision
Pavitha, N., & Sugave, S. R. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150237 · Cited by 8
“Guess what I'm doing”: Extending legibility to sequential decision tasks
Faria, M., Melo, F. S., & Paiva, A. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104107 · Cited by 12
“Most approaches to improve AIA transparency have focused on making more transparent technical aspects of AIA such as the reason behind application failures [36, 37, 38] or the inference processes of complex decision process like deep learning approaches [39, 40, 41].”
BackgroundDagani, J., Buizza, C., Ferrari, C., & Ghilardi, A. (2024) · Psicologia Reflexão e Crítica · doi:10.1186/s41155-024-00301-6 · Cited by 9
“Similarly, the interpretability of machine learning findings could be hard if not supported by preliminary hypotheses or validated by further analytical methods and techniques (Carvalho et al., 2019).”
BackgroundEvaluating Machine Learning Techniques for Enhanced Prediction of Building Energy Consumption
Vonitsanos, G., Kanavos, A., & Mylonas, P. (2024) · South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM) · doi:10.1109/seeda-cecnsm63478.2024.00018 · Cited by 8
“• Classification Metrics : Metrics such as Precision, Re-call, and F1-score assess the model’s ability to classify peak consumption periods correctly, crucial for effective energy management [5], [23].”
BackgroundEvaluating What Others Say: The Effect of Accuracy Assessment in Shaping Mental Models of AI Systems
Do, H. J., Brachman, M., Dugan, C., Pan, Q., Rai, P., Johnson, J. M., & Thawani, R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686912 · Cited by 12
“Researchers have worked to support users in building accurate mental models with explainable AI (XAI) and transparency in AI systems as surveyed in [1, 6, 18, 28].”
Accurate and Ubiquitous Floor Identification at the Edge using a Single Cell Tower
Mostafa, S., Youssef, M. A., & Harras, K. A. (2024) · IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec62691.2024.00024 · Cited by 7
“We plan to employ recent machine learning explainability methods [81] on our model to explain its predictions for better transparency.”
The disagreement dilemma in explainable AI: can bias reduction bridge the gap
Bhardwaj, N., & Parashar, G. (2025) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-025-02712-9 · Cited by 8
Ukwuoma, C. C., Cai, D., Bamisile, O., Bizi, A. M., Amos, T. J., Delali, F. L., Thomas, D., & Huang, Q. Z. (2025) · Renewable Energy · doi:10.1016/j.renene.2025.123076 · Cited by 7
Predicting non-shows at outpatient appointments in internal medicine using machine learning models
Osorio, F. O., Pedroza, S., Rebellón-Sánchez, D. E., Fernandez, R. F., Tabares-Soto, R., Bravo-Ortíz, M. A., & Cruz, G. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2762 · Cited by 9
“These measures quantify the accuracy and predictive ability of the model, allowing adjustments to enhance its performance ( Carvalho, Pereira & Cardoso, 2019 ).”
Explaining transformer-based next activity prediction by using attention scores
Käppel, M., Ackermann, L., Jablonski, S., & Härtl, S. (2025) · Process Science · doi:10.1007/s44311-025-00018-4 · Cited by 8
“The field of explainable artificial intelligence (XAI) aims to address this issue by developing techniques (so-called explainers ) that explain the decisions of machine learning models (Carvalho et al. 2019; Nauta et al. 2023).”
Intelligent System Using Data to Support Decision-Making
Anderková, V., Babič, F., Paraličová, Z., & Javorská, D. (2025) · Applied Sciences · doi:10.3390/app15147724 · Cited by 8
Prada, E. J. A., & Mosquera, C. D. C. (2025) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-025-01593-3 · Cited by 11
“The relationship between these two concepts often pres - ents as a trade-off [ 42].”
An exploratory study of interpretability for face presentation attack detection
Sequeira, A. F., Gonçalves, T., Silva, W., Pinto, J. R., & Cardoso, J. S. (2021) · IET Biometrics · doi:10.1049/bme2.12045 · Cited by 13
“Further efforts should be devoted to reducing subjectivity in the evaluation of explanations through objective metrics and procedures [48].”
BackgroundSelf-citationOliveira, F. R. D. S., & Neto, F. B. D. L. (2023) · IEEE Access · doi:10.1109/access.2023.3250262 · Cited by 9
“Carvalho et al. [24] also suggested that the explainability of AI systems must be considered in the context of the explanation needs, problem domain, and user types.”
BackgroundSuper-resolution and uncertainty estimation from sparse sensors of dynamical physical systems
Collins, A. M., Rivera-Casillas, P., Dutta, S., Cecil, O. M., Trautz, A. C., & Farthing, M. W. (2023) · Frontiers in Water · doi:10.3389/frwa.2023.1137110 · Cited by 9
“While there is burgeoning research on the explainability of neural networks (Goldstein et al., 2015; Carvalho et al., 2019; Nori et al., 2019; Apley and Zhu, 2020) (and many others), these results are often for simpler models and cases that are not in the scope of technological readiness for dynamical physical systems.”
BackgroundExplaining Deep Face Algorithms Through Visualization: A Survey
John, T. A., Balasubramanian, V. N., & Jawahar, C. V. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2023.3319837 · Cited by 8
The emergence and need for explainable AI
Chia, H. L. B. (2023) · Advances in Engineering Innovation · doi:10.54254/2977-3903/3/2023023 · Cited by 8
Interpretability Methods for Graph Neural Networks
Khan, A., & Mobaraki, E. B. (2023) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa60987.2023.10302600 · Cited by 15
Chen, Z., Dong, Z. Y., & Xu, Y. H. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3341244 · Cited by 7
“Interpreta-tiontechniquesaimtoincreasethetransparency,accountability, and reliability of ML models by providing explanations for the predictions they make, allowing stakeholders to understand how the model works and why it makes certain predictions [55], [56].”
BackgroundAlternative feature selection with user control
Bach, J., & Böhm, K. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00527-8 · Cited by 7
“Besides conventional feature selection, post hoc feature importance methods play a major role in the field of machine-learninginterpretability[9,48].”
Evaluative Item-Contrastive Explanations in Rankings
Castelnovo, A., Crupi, R., Mombelli, N., Nanino, G., & Regoli, D. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10311-2 · Cited by 10
Attention Please: What Transformer Models Really Learn for Process Prediction
Käppel, M., Ackermann, L., Jablonski, S., & Härtl, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70396-6_12 · Cited by 10
Bhatnagar, S., & Agrawal, R. (2024) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v13i6.8378 · Cited by 10
“Post-model techniques share post-hoc insights to find ways of improving the decision through yet unknown findings [3].”
MethodIdentificando evasão fiscal em empresas de fachada e em créditos ilegais de ICMS
Gomes, G. S. L., & Balaniuk, R. (2024) · Revista de Administração Pública · doi:10.1590/0034-761220230303 · Cited by 7
Ethical transparency in business failure prediction: uncovering the black box of xgboost algorithm
Martínez, M. R., Campillo, J. P., & Carmona, P. (2024) · Spanish Journal of Finance and Accounting / Revista Española de Financiación y Contabilidad · doi:10.1080/02102412.2024.2423498 · Cited by 8
“In response to these concerns, there has been a growing interest in developing XAI methods that can provide insights into how AI models make decisions (Carvalho et al., 2019).”
Implementation of generative adversarial networks in HPCC systems using GNN bundle
Karthik, A., Shetty, J., Shobha, G., & Dev, R. (2021) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v10.i2.pp374-381 · Cited by 7
“Machine learning evaluation metrics like [24] AUC, F1 score.”
MethodA Short Survey on Machine Learning Explainability: An Application to Periocular Recognition
Brito, J., & Proença, H. P. M. C. (2021) · Electronics · doi:10.3390/electronics10151861 · Cited by 7
“There are many other properties by which we can assess the usefulness of a given technique [1], but above all else, a technique must be able to make a black-box less opaque and much more user-friendly.”
BackgroundSERF: Interpretable Sleep Staging using Embeddings, Rules, and Features
Al-Hussaini, I., & Mitchell, C. S. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557700 · Cited by 10
“However, the lack of interpretability remains an obstacle to widespread adoption for a high stakes application like healthcare [6, 11, 15, 35, 40].”
BackgroundWhat Is Common to Transportation and Health in Machine Learning Education? The Domain Neglect Bias
Mike, K., & Hazzan, O. (2022) · IEEE Transactions on Education · doi:10.1109/te.2022.3218013 · Cited by 8
“Developing both interpretable models and explanation methods is currently one of the main efforts the ML research community is engaged in [7] and [8], as interpretability methods are required to help users build trust in ML models and understand their capabilities.”
MethodAttention-like feature explanation for tabular data
Konstantinov, A. V., & Utkin, L. V. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-022-00351-y · Cited by 9
“Comprehensive reviews providing details and peculiarities of most explanation methods can be found in survey papers [48, 49, 50, 51, 4, 52, 53, 54].”
BackgroundTeaching Factory Management in the Industrial Era 4.0 in Indonesia
Perwiranegara, A. A. (2022) · International Journal of Science and Society · doi:10.54783/ijsoc.v4i3.508 · Cited by 7
Wang, Z., Initiative, A. D. N., Wang, J., Liu, N., Liu, C., Li, X., Dong, L., Zhang, R., Mao, C., Duan, Z., et al. (2022) · Journal of Alzheimer s Disease · doi:10.3233/jad-220502 · Cited by 9
“However, the consistency between their interpretations and the original models is not guaranteed [23, 29].”
Robust Intrusion Detection System with Explainable Artificial Intelligence
Paltun, B. G., Fuladi, R., & Malki, R. E. (2025) · Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit) · doi:10.1109/eucnc/6gsummit63408.2025.11037121 · Cited by 6
Analyzing consumer IoT traffic from security and privacy perspectives: a comprehensive survey
Jia, Y., Song, Y., Liu, Z., Tan, Q., Song, Y., Zhang, Y., & Liu, Z. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-41155-0 · Cited by 7
Lazar, A., Pokhrel, P., & Das, S. R. (2026) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213026400038 · Cited by 6
Yu, K., Ishikura, S., Usukura, Y., Shigoku, Y., & Hayashi, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402338 · Cited by 6
“Carvalho et al. conducted a comprehensive survey of interpretability methods and underscored their relevance for trustworthy machine learning [9].”
Morer, F. E., Wittek, S., & Rausch, A. (2023) · Urban Water Journal · doi:10.1080/1573062x.2023.2282126 · Cited by 7
“…learning models to be interpretable and go beyond the black-box paradigm (Ribeiro, Singh, and Guestrin 2016), the rationale behind the predictions cannot be understood and the internal logic is not transparent to the user or analyst (Carvalho, Pereira, and Cardoso 2019; Guidotti et al. 2018).”
BackgroundXTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification
Höllig, J., Thoma, S., & Grimm, F. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00168 · Cited by 10
“There are no generally agreed upon-metrics measuring the quality of explanations, and comparisons between different implementations and metrics are difficult (e.g., [7]–[9]).”
BackgroundGoverning AI through interaction: situated actions as an informal mechanism for AI regulation
Papyshev, G. (2024) · AI and Ethics · doi:10.1007/s43681-024-00446-1 · Cited by 7
“These principles often encompass explainability and interpretability [5, 9, 16, 43], transparency [16, 51], usability [34, 48], human-in-the-loop considerations [3, 10, 19, 53], and human-centered AI [12, 26, 40].”
Wang, Y., Qiu, K., Telea, A. C., Hou, Z., Zhou, T., Cai, Y., Ding, Z., Yu, H., & Deng, J. (2024) · American Mineralogist · doi:10.2138/am-2023-9254 · Cited by 6
“The “black box” nature of machine models, however, hinders our understanding of decision-making processes during machine learning (Lipton 2018; Carvalho et al. 2019; Molnar 2020).”
The ontological quandary of deepfakes
Fasoro, S. A. (2024) · AI & Society · doi:10.1007/s00146-024-01902-6 · Cited by 7
“Yet Carvalho et al (2019) contends computational media reveal ontological instability at the core of all representation.”
Online Detection and Infographic Explanation of Spam Reviews with Data Drift Adaptation
de Arriba-Pérez, F., García-Méndez, S., Leal, F., Malheiro, B., & Burguillo, J. C. (2024) · Informatica · doi:10.15388/24-infor562 · Cited by 7
ARIA, HaRIA, and GeRIA: Novel Metrics for Pre-Model Interpretability
Pawlicki, M. (2024) · IEEE Access · doi:10.1109/access.2024.3454084 · Cited by 7
“…closely related terms such as ‘‘explainability,’’ ‘‘interpretability’’, ‘‘understandability’’, ‘‘comprehensibility’’, ‘‘intelligibility’’, and ‘‘transparency’’ [7], [8], in this paper the terms will be used interchangeably - so as not to take a side in the argument until the terms coagulate.”
BackgroundFischer, R. J., Rekabdarkolaee, H. M., Joshi, D. R., Clay, D. E., & Clay, S. A. (2024) · Agronomy Journal · doi:10.1002/agj2.21670 · Cited by 6
Interpretability in the modeling spectrum: A conceptual framework and a quantification index
Aguirre-Zapata, E., Álvarez, H., Lema-Pérez, L., di Sciascio, F. A., & Amicarelli, A. (2024) · Ecological Modelling · doi:10.1016/j.ecolmodel.2024.110882 · Cited by 6
Mouakher, A., & Kononov, R. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02421-y · Cited by 10
“2 The taxonomy of XAI according to [62] complexity and high-dimensional input.”
Stradowski, S., & Madeyski, L. (2024) · IEEE Software · doi:10.1109/ms.2024.3505544 · Cited by 6
“Also, not all of the available approaches, premodel, in-model, and postmodel, 11 can satisfy all the requirements with the desired level of performance.”
InfluentialBouasabah, M. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-75329-9_29 · Cited by 6
Knee Cartilage Degradation in the Medial and Lateral Anatomical Compartments: A Radiomics Study
Angelone, F., Ciliberti, F. K., Jónsson, H., Gíslason, M. K., Romano, M., Franco, A., Amato, F., & Gargiulo, P. (2024) · IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE) · doi:10.1109/metroxraine62247.2024.10796456 · Cited by 7
“…calculated as the harmonic mean of the precision and recall scores [33]; prevalence, i.e. the proportion of true positive cases in the dataset [32] and finally the balanced accuracy [34], the average of sensitivity and specificity, providing a balanced assessment of the model’s performance in…”
Xu, J., Mo, S., Chen, Z., Ke, H., & Jiang, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110071 · Cited by 6
Deng, F., Feng, C. H., Gao, N., & Zhang, L. (2025) · Transactions on Artificial Intelligence · doi:10.53941/tai.2025.100005 · Cited by 9
Mapstone, C., & Płusa, B. (2025) · Development · doi:10.1242/dev.202066 · Cited by 7
Moore, R. C., Gurchiek, R. D., & Avedesian, J. M. (2025) · Sports Engineering · doi:10.1007/s12283-025-00497-5 · Cited by 6
“While these benefit from improved interpretability relative to more flexible methods, they may not generalize beyond the study sample, which can include a large variety of pitchers and pitch types, and hence they may overstate model performance [22].”
Integration of AI based tools in dairy quality control: Enhancing pathogen detection efficiency
Devi, P., Karthikeyan, S., Giridhari, V. A., Dananjeyan, B., & Maruthamuthu, T. (2025) · Journal of Food Measurement & Characterization · doi:10.1007/s11694-025-03269-8 · Cited by 7
“The use of artificial intelligence in predictive maintenance can assist food processing companies improve their maintenance schedules, lowering the cost of unexpected break-downs and repairs [56].”
Canha, D., Kubler, S., Främling, K., & Fagherazzi, G. (2025) · ACM Computing Surveys · doi:10.1145/3737445 · Cited by 10
“4): This property assesses whether the explanation visualization omits or occludes the original input instance, which should be visible for explainee inspection [11].”
InfluentialLoureiro, J., Petrella, L., Gil, P. S., Rocha, T., Paredes, S., & Henriques, J. M. O. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3577486 · Cited by 8
“…performance indicator (positive or negative) of that system, such as accuracy, inaccuracy, availability, downtime rate, responsiveness, etc.” Another related definition of model’s reliability is proposed in [46]: “Ensure that small changes in the input do not cause large changes in the prediction”.”
When algorithms meet ethics: Mapping consumer moral responses to AI versus human marketing practices
tana, G., Chai, J., Chen, W., & Wauk, G. (2026) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2026.104729 · Cited by 7
Uncertainty Interpretation of the Machine Learning Survival Model Predictions
Utkin, L. V., Zaborovsky, V. S., Kovalev, M. S., Konstantinov, A. V., Politaeva, N. A., & Lukashin, A. (2021) · IEEE Access · doi:10.1109/access.2021.3108341 · Cited by 7
An Empirical Evaluation of Constrained Feature Selection
Bach, J., Zoller, K., Trittenbach, H., Schulz, K., & Böhm, K. (2022) · SN Computer Science · doi:10.1007/s42979-022-01338-z · Cited by 6
“Constraints also give way to interpretability, a recent and highly relevant topic in machine learning [7, 17].”
BackgroundInfluential
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.