Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Abstract
The most successful Machine Learning (ML) systems remain complex black boxes to end-users, and even experts are often unable to understand the rationale behind their decisions. The lack of transparency of such systems can have severe consequences or poor uses of limited valuable resources in medical diagnosis, financial decision-making, and in other high-stake domains. Therefore, the issue of ML explanation has experienced a surge in interest from the research community to application domains. While numerous explanation methods have been explored, there is a need for evaluations to quantify the quality of explanation methods to determine whether and to what extent the offered explainability achieves the defined objective, and compare available explanation methods and suggest the best explanation from the comparison for a specific task. This survey paper presents a comprehensive overview of methods proposed in the current literature for the evaluation of ML explanations. We identify properties of explainability from the review of definitions of explainability. The identified properties of explainability are used as objectives that evaluation metrics should achieve. The survey found that the quantitative metrics for both model-based and example-based explanations are primarily used to evaluate the parsimony/simplicity of interpretability, while the quantitative metrics for attribution-based explanations are primarily used to evaluate the soundness of fidelity of explainability. The survey also demonstrated that subjective measures, such as trust and confidence, have been embraced as the focal point for the human-centered evaluation of explainable systems. The paper concludes that the evaluation of ML explanations is a multidisciplinary research topic. It is also not possible to define an implementation of evaluation metrics, which can be applied to all explanation methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“They found that some properties (clarity, broadness, and completeness) are still in shortage of appropriate metrics, and so is the class of explanations that are example-based.”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
In the citing paper
“It is not possible to come up with quantitative metrics that are applicable for all interpretability methods [169].”
From Transparency of deep neural networks for medical image analysis: A review of interpretability methods (2022) · cites this paper as [169]
In the citing paper
“They conclude their survey by stating that”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [141]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2×
“One way to objectively evaluate explanations is to verify whether the explanation mechanism satisfies (or does not satisfy) certain axioms, or properties [34].”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“[39] explained an ensemble of models, such as neural networks with a single model effectively.”
BackgroundCausality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Explanation is also a domain-specific note, and there cannot be an all-purpose explanation [22], and different types of explanations might be useful.”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“The machine learning explainability has been reviewed thoroughly in recent years [10,14,17,23,28,31,32].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 4×
“…Main effect complexity [46] Level of (dis)agreement [82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity…”
BackgroundInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 4×
“The advantages of application-grounded and human-grounded evaluations are that they can provide direct and strong evidence of success of explanations [9].”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Zhou, J., and Chen, F. (2018). Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent, Springer.
- Lage, I., and Doshi-Velez, F. (2020, January 17). Human-in-the-Loop Learning of Interpretable and Intuitive Representations. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, Vienna, Austria.
- Becker, B., Kohavi, R., and Sommerfield, D. (2001). Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann Publishers Inc.
- Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Vaughan, J.W., and Wallach, H. (2019). Manipulating and Measuring Model Interpretability. arXiv.
- Yang, M., and Kim, B. (2019). Benchmarking Attribution Methods with Relative Feature Importance. arXiv, 1907.
Cited by
Items 101–200 of 676
Optimal Charging Station Placement and Scheduling for Electric Vehicles in Smart Cities
Alanazi, F., Alshammari, T. O., & Azam, A. M. (2023) · Sustainability · doi:10.3390/su152216030 · Cited by 43
“The results indicate that, these indicators play an important role in determining the optimal charging station placement as suggested by the previous study [57].”
MethodResultWhy we do need explainable AI for healthcare
Cinà, G., Röber, T. E., Goedhart, R., & Birbil, Ş. İ. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-025-00209-4 · Cited by 51
XAI in the Context of Predictive Process Monitoring: An Empirical Analysis Framework
Elkhawaga, G., Abu-Elkheir, M., & Reichert, M. (2022) · Algorithms · doi:10.3390/a15060199 · Cited by 48
Hupont, I., Tolan, S., Güneş, H., & Gómez, E. (2022) · Scientific Reports · doi:10.1038/s41598-022-14981-6 · Cited by 56
Benchmarking eXplainable AI - A Survey on Available Toolkits and Open Challenges
Le, P. Q., Nauta, M., Nguyen, V. B., Pathak, S., Schlötterer, J., & Seifert, C. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/747 · Cited by 61
“Follow-up work extends and refines this categorization [Zhou et al. , 2021; Mohseni et al. , 2021; Vilone and Longo, 2021; Lopes et al. , 2022; Nauta et al. , 2023].”
BackgroundKaur, S., Gupta, S., Singh, S., Koundal, D., & Zaguia, A. (2022) · Soft Computing · doi:10.1007/s00500-022-06805-6 · Cited by 39
“Performance based on the different confusion matrix parameters such as accuracy, recall, precision and F1-score (Fu¨rnkranz and Flach 2003; Zhou et al. 2021) has been discussed.”
InfluentialRetractedEmmert‐Streib, F., & Yli-Harja, O. P. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232113149 · Cited by 45
Explainable AI for Text Classification: Lessons from a Comprehensive Evaluation of Post Hoc Methods
Cesarini, M., Malandri, L., Pallucchini, F., Seveso, A., & Xing, F. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10325-w · Cited by 53
“Previous researches tend to agree that the main distinction is between objective versus human-centred metrics [13, 59].”
Understanding Relations Between Perception of Fairness and Trust in Algorithmic Decision Making
Zhou, J., Verma, S., Mittal, M., & Chen, F. (2021) · 2021 8th International Conference on Behavioral and Social Computing (BESC) · doi:10.1109/besc53957.2021.9635182 · Cited by 36
Self-citationEvaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks
Huang, Q., Xue, M., Huang, W., Zhang, H., Song, J., Jing, Y., & Song, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00192 · Cited by 66
“Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. prototype is consistent across images, as shown in Fig.”
BackgroundMethodZhang, X., Wang, Y., Jiang, Y., Pacella, C. B., & Zhang, W. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02793-9 · Cited by 38
Oblizanov, A., Shevskaya, N. V., Kazak, A., Rudenko, М., & Dorofeeva, A. A. (2023) · Applied System Innovation · doi:10.3390/asi6010026 · Cited by 44
Supporting High-Uncertainty Decisions through AI and Logic-Style Explanations
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584080 · Cited by 30
“In a more recent article, Zhou et al. [62] propose a taxonomy for XAI evaluation methods, further distinguishing two types of metrics: subjective and objective.”
BackgroundThe Role of Explainability in Collaborative Human-AI Disinformation Detection
Schmitt, V., Villa-Arenas, L. F., Feldhus, N., Meyer, J., Spang, R. P., & Möller, S. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659031 · Cited by 68
Leveraging explainable AI for informed building retrofit decisions: Insights from a survey
Leuthe, D., Mirlach, J., Wenninger, S., & Wiethe, C. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114426 · Cited by 37
“Since the term Explainability is not clearly defined [78], there is a plethora of different approaches to XAI in the literature [79], for instance, in the form of texts and visualizations, with contents of examples, feature relevancies, and simplifying surrogate models [32].”
Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents
Widyasari, R., Weyssow, M., Irsan, I. C., Ang, H. W., Liauw, F., Ouh, E. L., Shar, L. K., Kang, H. J., & Lo, D. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773256 · Cited by 47
“We also plan to consider the strengths and weaknesses of different LLMs to choose the one that best fits as the base model for each role-specific task.”
A nascent design theory for explainable intelligent systems
Herm, L.-V., Steinbach, T., Wanner, J., & Janiesch, C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00606-3 · Cited by 41
DeepFixCX : Explainable privacy‐preserving image compression for medical image analysis
Gaudio, A., Smailagic, A., Faloutsos, C., Mohan, S., Johnson, E., Liu, Y., Costa, P., & Campilho, A. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1495 · Cited by 45
Fan, J., Shar, L. K., Zhang, R., Liu, Z., Yang, W., Niyato, D. T., & Lam, K.-Y. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3653434 · Cited by 40
“Metrics such as ROC and AUC-ROC o ff er more robust assessments of classification performance under class-imbalanced conditions [108].”
Explainable AI: To Reveal the Logic of Black-Box Models
Chinu, & Bansal, U. (2023) · New Generation Computing · doi:10.1007/s00354-022-00201-2 · Cited by 40
“Jianlong Zhou et al. [105] discussed qualitative metrics for Model-based explanations, Attributes-based explanations, and example-based explanations.”
Explainable Image Classification: The Journey So Far and the Road Ahead
Kamakshi, V., & Krishnan, N. C. (2023) · AI · doi:10.3390/ai4030033 · Cited by 44
Elkhawaga, G., Elzeki, O. M., Abu-Elkheir, M., & Reichert, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3357041 · Cited by 37
“We refer interested readers to corresponding surveys on evaluation methods [23], [24].”
Mangold, A., Zietz, J., Weinhold, S., & Pannasch, S. (2026) · Advances in Human-Computer Interaction · doi:10.1155/ahci/8450102 · Cited by 33
Ogunseyi, T. B., Thiyagarajan, G., He, H., Bist, V., & Du, Z. (2026) · Sensors · doi:10.3390/s26020363 · Cited by 38
“…and stability, as shown in Table 4, provide insight into the technical aspects of these explanations, they fall short of capturing the crucial human element: how meaningful, understandable, and ultimately actionable the explanations are for human users [93,94] such as security professionals.”
Explainable-AI in Automated Medical Report Generation Using Chest X-ray Images
Ahmed, S. B., Solis-Oba, R., & Ilie, L. (2022) · Applied Sciences · doi:10.3390/app122211750 · Cited by 36
“An overview of metrics for evaluating explainability properties (i.e., clarity, breadth, parsimony, completeness and soundness) is discussed in [85].”
BackgroundWang, M., Shen, K., Tai, C., Zhang, Q., Yang, Z., & Guo, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0277352 · Cited by 32
“Since the LGBM model supports feature classification, it is unnecessary to transform and normalize the data in the preprocessing stage [24].”
MethodA Meta Survey of Quality Evaluation Criteria in Explanation Methods
Löfström, H., Hammar, K., & Johansson, U. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-07481-3_7 · Cited by 46
“– Performance is a group of criterion that, together with appropriate trust, is most often referred to in the surveys (see figure 3) as important to the quality of the explanations and denotes the level of correctness of the model [1, 5, 3, 18, 19, 17, 2, 9, 4, 10, 6, 21].”
BackgroundInfluentialWhat is Missing in XAI So Far? An Interdisciplinary Perspective
Schmid, U., & Wrede, B. (2022) · Künstliche Intell. · doi:10.1007/s13218-022-00786-2 · Cited by 39
“This property of an explanation is called faithfulness or fidelity [58, 94, 96].”
BackgroundCalibrated explanations: With uncertainty information and counterfactuals
Löfström, H., Löfström, T., Johansson, U., & Sönströd, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123154 · Cited by 43
Modern Smart Cities and Open Research Challenges and Issues of Explainable Artificial Intelligence
Sindiramutty, S. R., Tan, C. E., Tee, W. J., Lau, S. P., Balakrishnan, S., Kaur, S., Jazri, H., & Aslam, M. F. A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6361-1.ch015 · Cited by 34
Sun, J., Shi, W., Giuste, F., Vaghani, Y. S., Tang, L., & Wang, M. D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46493-2 · Cited by 35
Reconsidering the regulation of facial recognition in public spaces
Solarova, S., Podroužek, J., Mesarčík, M., Gavorník, A., & Bieliková, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00194-0 · Cited by 38
“On the other hand, there are also risks of fully transparent and explainable FRT, because higher interpretability and explainability could lead to higher vulnerability to security incidents [39].”
BackgroundAyoub, O., Cicco, N. D., Ezzeddine, F., Bruschetta, F., Rubino, R., Nardecchia, M., Milano, M., Musumeci, F., Passera, C., & Tornatore, M. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109466 · Cited by 35
Rasol, M., Schmidt, F., & Ientile, S. (2023) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2023.130567 · Cited by 30
Nematirad, R., Ardehali, M. M., Khorsandi, A., & Mahmoudian, A. (2024) · IEEE Access · doi:10.1109/access.2024.3358404 · Cited by 29
“Further, to evaluate the performance of the learning algorithms in this study, root mean square error (RMSE), mean squared error (MSE), and mean absolute error (MAE) are used [34].”
MethodChen, Y., Wang, G., Zhu, Y., Xie, C., & Uddin, G. S. (2024) · European Journal of Finance · doi:10.1080/1351847x.2024.2358940 · Cited by 30
“ML models are like ‘black boxes’ that severely hinder people from interpreting their predictions or decisions (Carvalho, Pereira, and Cardoso 2019; Minh et al.2022 ; Zhou et al.2021 ).”
Chadaga, K., Prabhu, S., Sampathila, N., Chadaga, R., Bhat, D., Sharma, A., & Swathi, K. S. (2024) · SLAS TECHNOLOGY · doi:10.1016/j.slast.2024.100129 · Cited by 29
A survey on methods and metrics for the assessment of explainability under the proposed AI Act
Sovrano, F., Sapienza, S., Palmirani, M., & Vitali, F. (2021) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia210342 · Cited by 27
“To do so, we collected all the papers cited in [4], re-classifying them.”
BackgroundMethodInfluentialNdao, M. L., Youness, G., Niang, N., & Saporta, G. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110144 · Cited by 27
XAI-KG: Knowledge Graph to Support XAI and Decision-Making in Manufacturing
Rožanec, J. M., Zajec, P., Kenda, K., Novalija, I., Fortuna, B., & Mladenić, D. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-79022-6_14 · Cited by 29
Smith, H., Sweeting, M. J., Morris, T. P., & Crowther, M. J. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00124-y · Cited by 26
Explainability of Predictive Process Monitoring Results: Can You See My Data Issues?
Elkhawaga, G., Abu-Elkheir, M., & Reichert, M. (2022) · Applied Sciences · doi:10.3390/app12168192 · Cited by 30
Schrills, T., & Franke, T. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588594 · Cited by 31
Elkhawaga, G., Elzeki, O. M., Abu-Elkheir, M., & Reichert, M. (2023) · Electronics · doi:10.3390/electronics12071670 · Cited by 31
Trivedi, S., Tran, T. A., Faruqui, N., & Hassan, M. M. (2023) · International Conference on Smart Computing and Application (ICSCA) · doi:10.1109/icsca57840.2023.10087713 · Cited by 27
“The performance of the models has been evaluated using the state-of-the-art machine learning evaluation metrics [44] accuracy, precision, recall, and F1-score, which are defined by equation 7, 8, 9, and 10, respectively, and listed in table V. Precision = TP TP + FP (8) The experimental result…”
MethodThrough the looking glass: evaluating post hoc explanations using transparent models
Velmurugan, M., Ouyang, C., Sindhgatta, R., & Moreira, C. (2023) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00445-1 · Cited by 27
“In particular, past evaluations of explanation fidelity , a measure to assess the correctness and completeness of an explanation with respect to the underlying model [59], are specific to particular classes of datasets or specific explainable methods.”
Chen, R., Wang, X., Liu, X., & Wang, S. (2024) · Forests · doi:10.3390/f15112024 · Cited by 25
AutoXAI: A Framework to Automatically Select the Most Adapted XAI Solution
Cugny, R., Aligon, J., Chevalier, M., Roman‐Jimenez, G., & Teste, O. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557247 · Cited by 44
“Thus, this type of evaluation is inexpensive, fast and can lead to a formal comparison of explanation methods Zhou et al. [2021].”
BackgroundMethodSaranti, A., Hudec, M., Mináriková, E., Takáč, Z., Großschedl, U., Koch, C., Pfeifer, B., Angerschmid, A., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4040047 · Cited by 28
“To overcome this issue, several quality indicators and constraints have recently been developed within the upcoming field of actionable explainable AI (AXAI) [7] (see, for example, [5,8] and see a survey on XAI methods [9] and a recent overview of methods of XAI [10]).”
BackgroundSelf-citationPrabhu, A., Morrison, S. M., Fox, P., Ma, X., Wong, M. L., Williams, J., McGuinness, K. N., Krivovichev, S. V., Lehnert, K. A., Ralph, J., et al. (2023) · American Mineralogist · doi:10.2138/am-2022-8613 · Cited by 28
“…of quality criteria (e.g., Prabhu et al. 2021b), but results from the data analysis may need to use quantitative metrics to evaluate results from a descriptive, prescriptive, or predictive model (e.g., Statnikov et al. 2008; Hossin and Sulaiman 2015; Tomašev and Radovanović 2016; Zhou et al. 2021).”
MethodA New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)
Speith, T., & Langer, M. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00061 · Cited by 35
Explingo: Explaining AI Predictions using Large Language Models
Zytek, A., Pidò, S., Alnegheimish, S., Berti‐Équille, L., & Veeramachaneni, K. K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825114 · Cited by 44
Parwez, K., Sohail, S. I., Parwez, F., & Zia, M. A. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101618 · Cited by 28
Mahmood, T., Naseem, S., Ashraf, R., Asif, M., Umair, M., & Shah, M. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08827-z · Cited by 26
“The results in terms of precision, recall, and F1-score are also shown in Fig.”
InfluentialFolgado, D., Barandas, M., Famiglini, L., Santos, R., Cabitza, F., & Gamboa, H. F. S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101955 · Cited by 28
Chari, S., Seneviratne, O. W., Ghalwash, M. F., Shirai, S. S., Gruen, D. M., Meyer, P., Chakraborty, P., & McGuinness, D. L. (2023) · Semantic Web · doi:10.3233/sw-233282 · Cited by 34
“Additionally, they focus more on integrating the outputs of logical reasoners to populate explanation types, instead of a wider breadth of AI explanation methods, such as those listed in Zhou et al. [6] and Arya et al. [7].”
BackgroundMethodInfluentialBai, S., Shi, S., Han, C., Yang, M., Gupta, B. B., & Arya, V. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.04.037 · Cited by 26
Weber, R. O., Johs, A. J., Goel, P., & Marques-Silva, J. P. (2024) · AI Magazine · doi:10.1002/aaai.12184 · Cited by 45
What Makes a Good Explanation?: A Harmonized View of Properties of ExplanationsPreprint
Chen, Z., Subhash, V., Havasi, M., Pan, W., & Doshi‐Velez, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.05667 · Cited by 37
“The identification of important properties in explanations by reviewing various explanation evaluation metrics in literature has been offered by Zhou et al. (2021).”
BackgroundMachine learning framework for predicting susceptibility to obesity
Shaban, W. M., Moustafa, H. E.-D., & El-Seddek, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-20505-9 · Cited by 22
“These models are Linear Regression (LR), LightGBM (LGBM), XGBoost (XGB), AdaBoost, Multi-Layer Perceptron (MLP), KNN, and SVM 40,41 : • Linear regression (LR): LR served as an efficient and basic model, well-suited for tasks involving binary and multiclass classification 42 .”
The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifiers
Nauta, M., & Seifert, C. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44064-9_21 · Cited by 30
dos Reis, M. Í., Gonçalves, J. N. C., Cortez, P. C., Carvalho, M. S., & Fernandes, J. M. (2024) · Computers in Industry · doi:10.1016/j.compind.2024.104233 · Cited by 26
Explainable artificial intelligence for molecular design in pharmaceutical research
Lamens, A., & Bajorath, J. (2026) · Chemical Science · doi:10.1039/d5sc08461j · Cited by 25
A quantitative evaluation of explainable AI methods using the depth of decision tree
AHMED, N. A. M., & Alpkoçak, A. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3924 · Cited by 22
“As a result, the depth of the decision tree can indicate how complex the XAI method is [7, 8, 15].”
BackgroundMethodRadiya, K., Joakimsen, H. L., Mikalsen, K. Ø., Aahlin, E. K., Lindsetmo, R., & Mortensen, K. E. (2023) · European Radiology · doi:10.1007/s00330-023-09609-w · Cited by 24
“In the studies like segmentation task, measuring parameter like mean surface distance with standard error should be reported to get overall transparency of the model performance [116].”
AI micro-decisions in FinTechs: a mixed method research design
Issa, H., Jabbouri, R., & Mehanna, R.-A. (2023) · Management Decision · doi:10.1108/md-10-2022-1336 · Cited by 24
Tamp-X: Attacking explainable natural language classifiers through tampered activations
Ali, H. M., Khan, M. S., Al‐Fuqaha, A., & Qadir, J. (2022) · Computers & Security · doi:10.1016/j.cose.2022.102791 · Cited by 22
Statistical arbitrage powered by Explainable Artificial Intelligence
Carta, S., Consoli, S., Podda, A. S., Recupero, D. R., & Stanciu, M. M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117763 · Cited by 26
Maamoun, I., Rushdi, M. A., Falyouna, O., Eljamal, R., & Eljamal, O. (2022) · Separation and Purification Technology · doi:10.1016/j.seppur.2022.122863 · Cited by 21
A Conceptual Model Framework for XAI Requirement Elicitation of Application Domain System
Aslam, M., Segura-Velandia, D., & Goh, Y. M. (2023) · IEEE Access · doi:10.1109/access.2023.3315605 · Cited by 21
“Some are measured using computational methods referred to as objective metrics [33], [62], [63]; while others rely on subjective methods based on users’ feedback [33].”
Zitouni, F., Almazyad, A. S., Xiong, G., Mohamed, A. W., & Harous, S. (2024) · IEEE Access · doi:10.1109/access.2024.3367440 · Cited by 20
“In classification tasks in ML, various evaluation metrics are used to assess the performance of model’s predictions [51].”
MethodSokol, K., & Vogt, J. E. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651047 · Cited by 35
“To this end, researchers have devised numerous algorithmic and (subjective or objective) user-based evaluation techniques as well as frameworks built on top of them, whose overviews, taxonomies and surveys are widely available [11, 16, 25, 45, 49, 54, 55, 67, 84].”
InfluentialKundu, R. K., & Hoque, K. A. (2023) · Annual Conference of the PHM Society · doi:10.36001/phmconf.2023.v15i1.3472 · Cited by 19
“A comprehensive survey of these metrics for evaluating explanation methods is available (Zhou, Gandomi, Chen, & Holzinger, 2021).”
MethodExplainable empirical risk minimization
Zhang, L., Karakasidis, G., Odnoblyudova, A., Dogruel, L., Tian, Y., & Jung, A. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-09269-3 · Cited by 23
“Moreover, there is no consensus about how to measure the explainability of a ‘‘simple’’ model [3].”
Pafla, M., Larson, K. S., & Hancock, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642934 · Cited by 24
Ghanvatkar, S., & Rajan, V. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3393719 · Cited by 19
“Evaluating Explanations From AI Algorithms for Clinical Decision-Making: A Social Science-Based Approach”
BackgroundMethodCalibrated explanations for regression
Löfström, T., Löfström, H., Johansson, U., Sönströd, C., & Matela, R. (2025) · Machine Learning · doi:10.1007/s10994-024-06642-8 · Cited by 22
“In recent times, Decision Support Systems (DSSs) in various domains such as retail, sport, or defence have been incorporating Artificial Intelligence (AI) extensively [1].”
Sheth, F., Chatter, I., Jasra, M., Kumar, G., & Sharma, R. (2025) · Plant Methods · doi:10.1186/s13007-025-01421-5 · Cited by 23
“These visualizations track learning progress over time, revealing trends that help detect overfitting or underfitting and thus deepen insights into learning stability, model behavior, and anticipated generalizability [92].”
How to Evaluate Explainability? - A Case for Three Criteria
Speith, T. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00024 · Cited by 21
Kares, F., Speith, T., Zhang, H., & Langer, M. (2026) · Computers in Human Behavior · doi:10.1016/j.chb.2026.109163 · Cited by 21
“Most studies evaluate a single XAI approach with one type of metric or restrict themselves to one family of evaluation [81, 88].”
InfluentialAn occupant-centered approach to improve both his comfort and the energy efficiency of the building
Boulmaiz, F., Reignier, P., & Ploix, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108970 · Cited by 19
Human-centric and semantics-based explainable event detection: a survey
Kolajo, T., & Daramola, O. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10525-0 · Cited by 22
“Future studies on human-centric XAI assessments should identify novel approaches that efficiently gather subjective measures to evaluate experimental user designs (Zhou et al. 2021 ).”
BackgroundGenerating multi-level explanations for process outcome predictions
Wickramanayake, B., Ouyang, C., Xu, Y., & Moreira, C. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106678 · Cited by 19
Pellano, K. N., Strümke, I., & Ihlen, E. A. F. (2024) · Sensors · doi:10.3390/s24061940 · Cited by 18
“In this work, we focus on the two primary metrics highlighted in [12]: faithfulness [13], which gauges how closely an explanation mirrors the model’s internal logic and stability [14], which pertains to the consistency of a model’s explanations across similar inputs.”
BackgroundWarnett, S. J., Ntentos, E., & Zdun, U. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112257 · Cited by 18
“J. Zhou et al. [39] conducted a survey offering an overview of methods proposed in the literature to assess ML model explanations.”
Song, Y., Li, C., Xing, W., Lyu, B., & Zhu, W. (2025) · The Internet and Higher Education · doi:10.1016/j.iheduc.2025.101000 · Cited by 20
Nor, A. K. B. M., Pedapati, S. R., & Muhammad, M. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03869 · Cited by 28
“[73] developed an overview of evaluation metrics and methods.”
MethodExplainable person–job recommendations: challenges, approaches, and comparative analysis
Tang, F., Zhu, R., Yao, F., Wang, J., Luo, L., & Li, B. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1660548 · Cited by 24
Lubis, A. R., Lase, Y. Y., Darwis, A. R., & Witarsyah, D. (2023) · Ingénierie des systèmes d information · doi:10.18280/isi.280522 · Cited by 17
“Evaluation metrics, essential tools in machine learning and data science, are utilized to assess the performance of a model or classification system in predicting classes within a specific dataset [36, 37].”
MethodKaur, H., Conrad, M. R., Rule, D., Lampe, C. A. C., & Gilbert, É. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637354 · Cited by 33
Evaluating Explainable Machine Learning Models for Clinicians
Scarpato, N., Nourbakhsh, A., Ferroni, P., Riondino, S., Roselli, M., Fallucchi, F., Barbanti, P., Guadagni, F., & Zanzotto, F. M. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10297-x · Cited by 22
Kiasari, M. M., & Aly, H. H. (2024) · Fire · doi:10.3390/fire7090296 · Cited by 18
Sanida, T., Sanida, M. V., Sideris, A., & Dasygenis, M. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4030109 · Cited by 17
Cancer pain detection based on physiological parameters and machine learning
Tashtoush, N., Al-Ghraibah, A. M., Sulaiman, S. A. S., AMRAN, F. A., Rahim, S. K., & Harun, S. N. (2024) · Cogent Engineering · doi:10.1080/23311916.2024.2431581 · Cited by 17
“…it measures how well predictions are ranked, rather than their absolute values, and it is classification-threshold-invariant, which means that it measures the quality of the model ’ s predictions irrespective of the classification threshold (Dangeti, 2017; Steurer et al., 2021; Zhou et al., 2021).”
Evaluating Explainable AI Methods in Deep Learning Models for Early Detection of Cerebral Palsy
Pellano, K. N., Strümke, I., Groos, D., Adde, L., & Ihlen, E. A. F. (2025) · IEEE Access · doi:10.1109/access.2025.3525571 · Cited by 18
“Fidelity or faithfulness [21, 22, 23] checks whether an explanation accurately identifies the features that influence a model’s predictions.”
BackgroundExplainable Bayesian Optimization
Chakraborty, T., Wirth, C., & Seifert, C. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08324-1_3 · Cited by 20
Belaid, M. K., Bornemann, R., Rabus, M., Krestel, R., & Hüllermeier, E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_5 · Cited by 19
MLcps: machine learning cumulative performance score for classification problems
Akshay, A., Abedi, M., Shekarchizadeh, N., Burkhard, F. C., Katoch, M., Bigger-Allen, A., Adam, R. M., Monastyrskaya, K., & Gheinani, A. H. (2022) · GigaScience · doi:10.1093/gigascience/giad108 · Cited by 16
An improved algorithm based on YOLOv5 for detecting Ambrosia trifida in UAV images
Xiaoming, C., Tianzeng, C., Haomin, M., Ziqi, Z., Dehua, W., Jianchao, S., & Wang, J. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1360419 · Cited by 18
Intrusion Detection in Wireless Sensor Networks using Machine Learning
Fares, H., Vibhute, A. D., Mouniane, Y., & Bouijij, H. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.01.052 · Cited by 16
Metadata
- Authors
- Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A.
- Affiliations
- University of Technology Sydney (AU)
- Medical University of Graz (AT)
- Corresponding author
- Zhou, Jianlong
- Gandomi, Amir H.
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2021-03-04
- Volume
- 10
- Issue
- 5
- Pages
- 593
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W3133543405
- Cited by
- 606
- References
- 80
- Citations by year
- 2026: 75 · 2025: 138 · 2024: 179 · 2023: 109 · 2022: 84 · 2021: 20
- FWCI
- 34.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University of Technology Sydney
- Austrian Science Fund (P 32554)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable machine learning, explanation quality evaluation, evaluation metrics, explainability, model-based explanation, example-based explanations, interpretability, fidelity, human-centered evaluation, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.