Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Abstract
The most successful Machine Learning (ML) systems remain complex black boxes to end-users, and even experts are often unable to understand the rationale behind their decisions. The lack of transparency of such systems can have severe consequences or poor uses of limited valuable resources in medical diagnosis, financial decision-making, and in other high-stake domains. Therefore, the issue of ML explanation has experienced a surge in interest from the research community to application domains. While numerous explanation methods have been explored, there is a need for evaluations to quantify the quality of explanation methods to determine whether and to what extent the offered explainability achieves the defined objective, and compare available explanation methods and suggest the best explanation from the comparison for a specific task. This survey paper presents a comprehensive overview of methods proposed in the current literature for the evaluation of ML explanations. We identify properties of explainability from the review of definitions of explainability. The identified properties of explainability are used as objectives that evaluation metrics should achieve. The survey found that the quantitative metrics for both model-based and example-based explanations are primarily used to evaluate the parsimony/simplicity of interpretability, while the quantitative metrics for attribution-based explanations are primarily used to evaluate the soundness of fidelity of explainability. The survey also demonstrated that subjective measures, such as trust and confidence, have been embraced as the focal point for the human-centered evaluation of explainable systems. The paper concludes that the evaluation of ML explanations is a multidisciplinary research topic. It is also not possible to define an implementation of evaluation metrics, which can be applied to all explanation methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“They found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
In the citing paper
“It is not possible to come up with quantitative metrics that are applicable for all interpretability methods [169].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [169]
In the citing paper
“They conclude their survey by stating that”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2×
“One way to objectively evaluate explanations is to verify whether the explanation mechanism satisfies (or does not satisfy) certain axioms, or properties [34].”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“[39] explained an ensemble of models, such as neural networks with a single model effectively.”
BackgroundCausality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Explanation is also a domain-specific note, and there cannot be an all-purpose explanation [22], and different types of explanations might be useful.”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“The machine learning explainability has been reviewed thoroughly in recent years [10,14,17,23,28,31,32].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 4×
“…Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity…”
BackgroundInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 4×
“The advantages of application-grounded and human-grounded evaluations are that they can provide direct and strong evidence of success of explanations [9].”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Zhou, J., and Chen, F. (2018). Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent, Springer.
- Lage, I., and Doshi-Velez, F. (2020, January 17). Human-in-the-Loop Learning of Interpretable and Intuitive Representations. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, Vienna, Austria.
- Becker, B., Kohavi, R., and Sommerfield, D. (2001). Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann Publishers Inc.
- Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Vaughan, J.W., and Wallach, H. (2019). Manipulating and Measuring Model Interpretability. arXiv.
- Yang, M., and Kim, B. (2019). Benchmarking Attribution Methods with Relative Feature Importance. arXiv, 1907.
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
“A summary of many quantitative metrics for the evaluation of explainability properties (i.e., clarity, broadness, parsimony, completeness, and soundness) for different explanation types, is presented in the work of Zhou et al. [141].”
BackgroundLongo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“Furthermore, XAI has evolved into an exceedingly multidisciplinary, interdisciplinary, and transdisciplinary field.”
BackgroundSelf-citationNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“[314] that presents a concise overview and discussion regarding evaluating XAI without a structured literature review, and the recent work of Vilone and Longo [263] that present a list of concepts related to explainability and discuss evaluation methods in combination with explainable AI methods.”
BackgroundInfluentialSalahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Explainable artificial intelligence (XAI) in finance: a systematic literature review
Černevičienė, J., & Kabašinskas, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10854-8 · Cited by 289
“The comprehensive evaluation of XAI methods, encompassing numerous techniques, procedures, and performance measurements, has been investigated and reported in published surveys such as Doshi-Velez and Kim (2017) and Zhou et al. (2021).”
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
Speith, T. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534639 · Cited by 337
“As a research discipline, XAI has grown exponentially in recent years, with annual publications easily exceeding several hundreds [10, 75].”
BackgroundSchwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328
“A full focus on metrics for evaluating XAI methods is set in the recent work [151] by Zhou et al.”
BackgroundMethodThe Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Krishna, S., Han, T., Gu, A., Jabbari, S., Wu, Z. S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2963888/v1 · Cited by 201
Research and application of machine learning for additive manufacturing
Qin, J., Hu, F., Liu, Y., Witherell, P., Wang, C. C. L., Rosen, D. W., Simpson, T. W., Lu, Y., & Tang, Q. (2022) · Additive Manufacturing · doi:10.1016/j.addma.2022.102691 · Cited by 391
“Although the evaluation metrics are not included in Table 5, many more details are reported in the ML literature [211,212].”
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
“This method involves important XAI technical, objective, and quantifiable metrics such as fidelity, consistency, robustness, e ffi ciency, and other dimensions [218].”
MethodChou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302
A Survey of Algorithmic Recourse: Contrastive Explanations and Consequential Recommendations
Karimi, A.-H., Barthe, G., Schölkopf, B., & Valera, I. (2023) · ACM Computing Surveys · doi:10.1145/3527848 · Cited by 301
“9Since the manuscript was irst published online, several works have pushed in this direction, including [58, 58, 106, 192, 251], and also a public repository for running many recourse generating methods in a uniied and comparable fashion [181].”
BackgroundMoustafa, N., Koroniotis, N., Keshk, M., Zomaya, A. Y., & Tari, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3280465 · Cited by 266
“It is also impossible to evaluate all XAI methods by a defined evaluation metric [209].”
MethodFour principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Machine Learning Toxicity Prediction: Latest Advances by Toxicity End Point
Cavasotto, C. N., & Scardino, V. (2022) · ACS Omega · doi:10.1021/acsomega.2c05693 · Cited by 213
“26,27 Moreover, one of the main challenges that computational toxicology faces today is the ability to obtain a mechanistic explanation or understanding of the identified toxic responses.”
BackgroundPost-hoc vs ante-hoc explanations: xAI design guidelines for data scientists
Retzlaff, C. O., Angerschmid, A., Saranti, A., Schneeberger, D., Röttger, R., Müller, H., & Holzinger, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101243 · Cited by 194
“Refer to Zhou et al. (2021) for a more exhaustive description and comparison of different evaluation factors for xAI methods.”
Self-citationShaik, T., Tao, X., Li, L., Xie, H., & Velásquez, J. D. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102040 · Cited by 196
“Evaluation can involve assessing classification accuracy, regression performance, clustering quality, or other domain-specific evaluation criteria [68].”
BackgroundRetzlaff, C. O., Das, S., Wayllace, C., Mousavi, P., Afshari, M. P., Yang, T., Saranti, A., Angerschmid, A., Taylor, M. E., & Holzinger, A. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15348 · Cited by 281
Self-citationKiasari, M., Ghaffari, M., & Aly, H. (2024) · Energies · doi:10.3390/en17164128 · Cited by 176
A Survey of Explainable Artificial Intelligence for Smart Cities
Javed, A. R., Ahmed, W., Pandya, S., Maddikunta, P. K. R., Alazab, M., & Gadekallu, T. R. (2023) · Electronics · doi:10.3390/electronics12041020 · Cited by 193
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207
Explainable Artificial Intelligence-Based Decision Support Systems: A Recent Review
Kostopoulos, G., Davrazos, G., & Kotsiantis, S. (2024) · Electronics · doi:10.3390/electronics13142842 · Cited by 151
Zero Touch Management: A Survey of Network Automation Solutions for 5G and 6G Networks
Coronado, E., Behravesh, R., Subramanya, T., Fernandez-Fernandez, A., Siddiqui, M. S., Costa-Perez, X., & Riggio, R. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3212586 · Cited by 183
“Metrics of different natures have been proposed to quantify XAI [283].”
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Llorca, D. F., Yela, D. F., Junklewitz, H., Scalzo, S., Mazzini, G., Sánchez, I., Garrido, J. S., et al. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594069 · Cited by 270
“The lack of a standardized set of metrics further complicates the technical evaluation of AI explanations [53, 72, 83, 123].”
BackgroundFrom Blackbox to Explainable AI in Healthcare: Existing Tools and Case Studies
Srinivasu, P. N., Sandhya, N., Jhaveri, R. H., & Raut, R. (2022) · Mobile Information Systems · doi:10.1155/2022/8167821 · Cited by 171
“*e toolkits are efficient in evaluating the explainability and understandability of the model through various evaluation metrics [39], which are categorized as Rationale explanation, Data explanation, and Safety and performance explanation [40].”
BackgroundA manifesto on explainability for artificial intelligence in medicine
Combi, C., Amico, B., Bellazzi, R., Holzinger, A., Moore, J. H., Zitnik, M., & Holmes, J. H. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102423 · Cited by 163
Self-citationA Comprehensive Survey of Deep Learning Approaches in Image Processing
Τρίγκα, Μ., & Dritsas, E. (2025) · Sensors · doi:10.3390/s25020531 · Cited by 207
Explainable Predictive Maintenance: A Survey of Current Methods, Challenges and Opportunities
Cummins, L., Sommers, A., Ramezani, S. B., Mittal, S., Jabour, J. E., Seale, M. A., & Rahimi, S. (2024) · IEEE Access · doi:10.1109/access.2024.3391130 · Cited by 200
“While several researchers argue that the terms are synonymous, viewing them as interchangeable to simplify discussions [8], [9], [10], [11], others assert that they capture distinct concepts [12], [13], [14], [15], [16], [17], [18], [19].”
BackgroundFairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161
Self-citationExplainable AI in Manufacturing and Industrial Cyber–Physical Systems: A Survey
Moosavi, S., Farajzadeh-Zanjani, M., Razavi‐Far, R., Palade, V., & Saif, M. (2024) · Electronics · doi:10.3390/electronics13173497 · Cited by 190
Herm, L.-V., Heinrich, K., Wanner, J., & Janiesch, C. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102538 · Cited by 200
“Especially in intelligent systems that use ML algorithms as a basis for decision making, a plethora of individual factors like self-efficacy, general distrust, or neuroticism can influence the view of the tradeoff (Mahmud et al., 2022; Zhou et al., 2021).”
BackgroundExplainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review
Giuste, F., Shi, W., Zhu, Y., Naren, T., Isgut, M., Sha, Y., Li, T., Gupte, M., & Wang, M. D. (2022) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3185953 · Cited by 162
“Based on the previous work [97], a guideline for evaluating XAI methods from both model behavior and human understanding perspectives is proposed and illustrated in Fig.”
MethodA Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139
“Reference [224] performs a survey on the quality evaluation of machine learning explanations.”
BackgroundAyano, Y. M., Schwenker, F., Dufera, B. D., & Debelee, T. G. (2022) · Diagnostics · doi:10.3390/diagnostics13010111 · Cited by 167
“Thus, examining the feature effects requires human expert intervention to determine the explanation’s clarity and soundness [166].”
BackgroundMethodSupervised Machine Learning Models for Liver Disease Risk Prediction
Dritsas, E., & Τρίγκα, Μ. (2023) · Computers · doi:10.3390/computers12010019 · Cited by 157
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124
Obaido, G., Mienye, I. D., Egbelowo, O. F., Emmanuel, I. D., Ogunleye, A., Ogbuokiri, B., Mienye, P., & Aruleba, K. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100576 · Cited by 154
Dikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112
“Zhou et al. [10] and Vilone and Longo [1] state that the “subjective” term requires the involvement of a human in the explanation’s evaluation, while the “objective” term concerns only evaluation aspects that do not depend on the judgements of the participants.”
BackgroundMethodInfluentialExplainable long-term building energy consumption prediction using QLattice
Wenninger, S., Kaymakci, C., & Wiethe, C. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.118300 · Cited by 123
Machine Learning: Models, Challenges, and Research Directions
Khoei, T. T., & Kaabouch, N. (2023) · Future Internet · doi:10.3390/fi15100332 · Cited by 120
Personas for Artificial Intelligence (AI) an Open Source Toolbox
Holzinger, A., Kargl, M., Kipperer, B., Regitnig, P., Plass, M., & Müller, H. (2022) · IEEE Access · doi:10.1109/access.2022.3154776 · Cited by 127
“However, since the perceived quality of an explanation is strongly dependent on the context and the user [41], it is crucial that these explanations are created with the users in mind to be understandable and useful.”
BackgroundSelf-citationMustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 117
Bhagat, S. K., Pilario, K. E., Babalola, O. E., Tiyasha, T., Yaqub, M., Onu, C. E., Pyrgaki, K., Falah, M. W., Jawad, A. H., Yaseen, D. A., et al. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.135522 · Cited by 100
A novel hybrid feature selection and ensemble-based machine learning approach for botnet detection
Hossain, M. A., & Islam, M. S. (2023) · Scientific Reports · doi:10.1038/s41598-023-48230-1 · Cited by 93
“A random model with an AUC of 0.5 is no better than guessing 64 .”
MethodInfluentialCaldwell, S., Sweetser, P., O’Donnell, N., Knight, M. J., Aitchison, M., Gedeon, T., Johnson, D. M., Brereton, M. F., Gallagher, M. R., & Conroy, D. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514257 · Cited by 109
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Self-citationNor, A. K. M., Pedapati, S. R., Muhammad, M. B., & Leiva, V. (2021) · Sensors · doi:10.3390/s21238020 · Cited by 86
“An overview of explanation metrics and methods is presented in [51], whereas the effectiveness of explanation from”
MethodGoing Beyond XAI: A Systematic Survey for Explanation-Guided Learning
Gao, Y., Gu, S., Jiang, J., Hong, S.-S., Yu, D., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3644073 · Cited by 121
“Along the line of interpretability techniques, many recent surveys also review the methods and metrics for comprehensively validating the quality of the explanation generated by the XAI techiniques [62, 100, 175].”
MethodZero-touch networks: Towards next-generation network automation
Rajab, M. E., Yang, L., & Shami, A. A. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110294 · Cited by 96
Hamida, S. U., Chowdhury, M. J. M., Chakraborty, N. R., Biswas, K., & Sami, S. K. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110149 · Cited by 95
Time series big data: a survey on data stream frameworks, analysis and algorithms
Almeida, A., Brás, S., Sargento, S., & Pinto, F. C. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00760-1 · Cited by 99
Vergaray, A. D., Rueda, N. D. G., Sánchez, M. S. A., Robles-Espiritu, W., & Quiñones, M. E. C. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100267 · Cited by 108
Li, J., Zhao, T. J., Yang, Q., Du, S., & Xu, L. (2024) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2024.105278 · Cited by 96
Pawlicki, M., Pawlicka, A., Uccello, F., Szelest, S., D’Antonio, S., Kozik, R., & Choraś, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128282 · Cited by 88
Subramanian, H. V., Canfield, C., & Shank, D. B. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102780 · Cited by 84
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
Dai, J. C., Upadhyay, S., Aïvodji, U., Bach, S. H., & Lakkaraju, H. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534159 · Cited by 127
“Prior research has studied several notions of explanation quality, namely, fidelity or faithfulness, stability, consistency, and sparsity [48, 55, 65, 80].”
BackgroundInfluentialExploring Evaluation Methods for Interpretable Machine Learning: A Survey
Alangari, N., Menaï, M. E. B., Mathkour, H. I., & Almosallam, I. A. (2023) · Information · doi:10.3390/info14080469 · Cited by 91
Metrics, Explainability and the European AI Act Proposal
Sovrano, F., Sapienza, S., Palmirani, M., & Vitali, F. (2022) · J · doi:10.3390/j5010010 · Cited by 71
“The main distinction between our work and [8] is probably our assumption that multiple definitions of explainability exist, each one possibly requiring its own type of metrics.”
BackgroundMethodInfluentialBernal, J., & Mazo, C. (2022) · Applied Sciences · doi:10.3390/app122010228 · Cited by 70
Chen, T., Wu, H., & Chiu, M.-C. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.111183 · Cited by 76
Gongane, V. U., Munot, M. V., & Anuse, A. D. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00248-9 · Cited by 90
“Subjective measures like trust and confidence are considered as important for human-centric evaluations [55].”
InfluentialNaveed, S., Stevens, G., & Robin-Kern, D. (2024) · Applied Sciences · doi:10.3390/app142311288 · Cited by 88
Hettikankanamage, N. D., Shafiabady, N., Chatteur, F., Wu, M., Din, F. U., & Zhou, J. J. (2025) · Sensors · doi:10.3390/s25216649 · Cited by 68
Self-citationBamal, A., Uddin, M. G., & Olbert, A. I. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e37073 · Cited by 73
“This comparison is conducted using various evaluation metrics, including accuracy, mean absolute error, root mean square error, mean square error, correlation, etc. [56,57].”
MethodLLMs for XAI: Future Directions for Explaining ExplanationsPreprint
Zytek, A., Pidò, S., & Veeramachaneni, K. K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.06064 · Cited by 67
“[21] to evaluate XAI explanations ( R1 ).”
BackgroundIntrusion Detection Systems: A State-of-the-Art Taxonomy and Survey
Alkasassbeh, M. S., & Baddar, S. W. A.-H. (2022) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-022-07412-1 · Cited by 70
“There is a wealth of literature on these and other metrics [7,8].”
BackgroundMachine Learning Technologies for Big Data Analytics
Gandomi, A. H., Chen, F., & Abualigah, L. (2022) · Electronics · doi:10.3390/electronics11030421 · Cited by 60
Self-citationExplaining AI through mechanistic interpretability
Kästner, L., & Crook, B. (2024) · European Journal for Philosophy of Science · doi:10.1007/s13194-024-00614-4 · Cited by 78
“…the divide-and-conquer strategy has been and continues to be productive, both in terms of novel methods and conceptual analysis (e.g., Barredo Arrieta et al., 2020; Belle & Papantonis, 2021; Köhl et al., 2019; Langer et al., 2021; Ribeiro et al., 2016; Sokol & Flach, 2020; Zhou et al., 2021).”
InfluentialVisser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
“Notably, Zhou et al. (2021) refer to fidelity and faithfulness as being the same concept, whereas Nauta et al. (2022) use the term correctness to reflect an explanations faithfulness with respect to the underlying model while Zhou et al. (2021) use the term soundness to refer to the correctness of…”
Enhanced Hand Gesture Recognition with Surface Electromyogram and Machine Learning
Kadavath, M. R. K., Nasor, M., & Imran, A. (2024) · Sensors · doi:10.3390/s24165231 · Cited by 58
“While accuracy is a straightforward and widely used metric, it can be misleading in the case of imbalanced datasets where one class significantly outnumbers the other [28,29].”
BackgroundSelf-eXplainable AI for medical image analysis: A survey and new outlooks
Hou, J., Liu, S., Bie, Y., Wang, H., Tan, A., Luo, L., & Chen, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104791 · Cited by 66
Towards Explainability for AI Fairness
Zhou, J. J., Chen, F., & Holzinger, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_18 · Cited by 81
Self-citationAyoub, O., Troìa, S., Andreoletti, D., Bianco, A. G., Tornatore, M., Giordano, S., & Rottondi, C. (2022) · Journal of Optical Communications and Networking · doi:10.1364/jocn.470812 · Cited by 62
Hulayyil, S. B., Li, S., & Xu, L. (2023) · Electronics · doi:10.3390/electronics12183927 · Cited by 67
“- Model evaluation: This aims to evaluate the trained model using various metrics with predicted models to assess the quality of the ML model, such as calculating the accuracy, precision, recall, and F1 in these algorithms to ensure the effectiveness [96].”
MethodEl-Khalek, A. A. A., Balaha, H. M., Alghamdi, N. S., Ghazal, M. A., Khalil, A. T., Abo-Elsoud, M. E. A., & El-Baz, A. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-52131-2 · Cited by 60
“It is important for evaluating model performance and deriving other metrics such as accuracy and F1 65–67 .”
How well can a large language model explain business processes as perceived by users?
Fahland, D., Fournier, F., Limonad, L., Skarbovsky, I., & Swevels, A. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102416 · Cited by 61
Wang, Q., Sun, T., & Li, R. (2025) · Sustainable Development · doi:10.1002/sd.70388 · Cited by 67
Transfer Learning-Based Automatic Hurricane Damage Detection Using Satellite Images
Kaur, S., Gupta, S., Singh, S., Hoang, V. T., Almakdi, S., Alelyani, T., & Shaikh, A. (2022) · Electronics · doi:10.3390/electronics11091448 · Cited by 53
Saifullah, S., Mercier, D., Lucieri, A., Dengel, A., & Ahmed, S. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1236947 · Cited by 61
“Tobacco3482 , 3 is another popular but small-scale dataset with 3,482 labeled document images.”
BackgroundValente, F., Paredes, S., Henriques, J. M. O., Rocha, T., Carvalho, P., & Morais, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00821-8 · Cited by 62
“Two main factors determine the explainability of an ML model: the parameters of the model and the human’s capacity to understand it (Zhou et al. 2021).”
BackgroundExplainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 61
“1 3 software systems on the explanation level [68, 69].”
BackgroundMethodAnomaly diagnosis of connected autonomous vehicles: A survey
Fang, Y., Min, H., Wu, X., Wang, W., Zhao, X., Martínez‐Pastor, B., & Teixeira, R. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102223 · Cited by 56
Wickramanayake, B., He, Z., Ouyang, C., Moreira, C., Xu, Y., & Sindhgatta, R. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108773 · Cited by 50
Explainability in process outcome prediction: Guidelines to obtain interpretable and faithful models
Stevens, A., & Smedt, J. D. (2023) · European Journal of Operational Research · doi:10.1016/j.ejor.2023.09.010 · Cited by 52
“For more information about the different XAI definitions, properties and/or metrics, we refer to [7], [9] and [10].”
BackgroundResultInfluentialPLENARY: Explaining black-box models in natural language through fuzzy linguistic summaries
Kaczmarek‐Majer, K., Casalino, G., Castellano, G., Dominiak, M., Hryniewicz, O., Kamińska, O., Vessio, G., & Díaz-Rodríguez, N. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.010 · Cited by 62
Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI)
Crook, B., Schlüter, M., & Speith, T. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00060 · Cited by 73
Wu, P., Tian, E., Tao, H., & Chen, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109756 · Cited by 58
Challenges and efforts in managing AI trustworthiness risks: a state of knowledge
Polemi, N., Praça, I., Kioskli, K., & Bécue, A. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1381163 · Cited by 46
A survey of Internet of medical things: technology, application and future directions
He, P., Huang, D., Wu, D., He, H., Wei, Y., Cui, Y., Wang, R., & Peng, L. (2024) · Digital Communications and Networks · doi:10.1016/j.dcan.2024.11.013 · Cited by 56
Towards Explainable Deep Neural Networks for the Automatic Detection of Diabetic Retinopathy
Alghamdi, H. S. (2022) · Applied Sciences · doi:10.3390/app12199435 · Cited by 47
“Perform Grad-CAM analysis to provide human-interpretable explanations of the deep architectures’ predictions of DR.”
MethodDeveloping a Fidelity Evaluation Approach for Interpretable Machine LearningPreprint
Velmurugan, M., Ouyang, C., Moreira, C., & Sindhgatta, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08492 · Cited by 53
“A key measure of functionally-grounded explanation fitness is explanation fidelity, which assesses the correctness and completeness of the explanation with respect to the underlying black box predictive model (Zhou et al, 2021).”
BackgroundEffects of AI and Logic-Style Explanations on Users’ Decisions Under Different Levels of Uncertainty
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588320 · Cited by 52
“Zhou et al. [89] propose a comprehensive overview of the state-of-the-art methods for the evaluation of machine learning explanations.”
BackgroundExplainability and vision foundation models: A survey
Kazmierczak, R., Berthier, É., Frehse, G., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103184 · Cited by 61
The effect of machine learning explanations on user trust for automated diagnosis of COVID-19
Goel, K., Sindhgatta, R., Kalra, S., Goel, R., & Mutreja, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105587 · Cited by 44
“In this work, we aim to generate ML explanations using existing algorithms and conduct a human-centred evaluation systematically via an application-grounded evaluation [6, 7].”
BackgroundMethodEvaluation Metrics in Explainable Artificial Intelligence (XAI)
Coroama, L., & Groza, A. P. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-20319-0_30 · Cited by 45
Machine Learning in Wear Prediction
Shah, R., Pai, N., Thomas, G., Jha, S., Mittal, V., Shirvani, K. A., & Liang, H. (2024) · Journal of Tribology · doi:10.1115/1.4066865 · Cited by 42
Metadata
- Authors
- Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A.
- Affiliations
- University of Technology Sydney (AU)
- Medical University of Graz (AT)
- Corresponding author
- Zhou, Jianlong
- Gandomi, Amir H.
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2021-03-04
- Volume
- 10
- Issue
- 5
- Pages
- 593
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W3133543405
- Cited by
- 606
- References
- 80
- Citations by year
- 2026: 75 · 2025: 138 · 2024: 179 · 2023: 109 · 2022: 84 · 2021: 20
- FWCI
- 34.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University of Technology Sydney
- Austrian Science Fund (P 32554)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable machine learning, explanation quality evaluation, evaluation metrics, explainability, model-based explanation, example-based explanations, interpretability, fidelity, human-centered evaluation, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.