Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 501–600 of 1,007
The impact of AI explanations on clinicians’ trust and diagnostic accuracy in breast cancer
Rezaeian, O., Asan, O., & Bayrak, A. E. (2025) · Applied Ergonomics · doi:10.1016/j.apergo.2025.104577 · Cited by 30
The Coming of Age of Interpretable and Explainable Machine Learning Models
Lisboa, P. J. G., Saralajew, S., Vellido, A., & Villmann, T. (2021) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2021.es2021-2 · Cited by 27
“A possible approach is to take an application-grounded evaluation on the respective task [10].”
BackgroundMethodGoštautaitė, D., & Sakalauskas, L. L. (2022) · Applied Sciences · doi:10.3390/app12115396 · Cited by 23
Explainable machine learning multi-label classification of Spanish legal judgements
de Arriba-Pérez, F., García-Méndez, S., González-Castaño, F. J., & González‐González, J. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.10.015 · Cited by 33
“…(such as graph-based (Cai et al., 2022), model agnostic (Leal et al., 2022), word embeddings (Qureshi & Greene, 2019) and visual explanations (Carvalho et al., 2019; Ye et al., 2021)), we pay special attention to natural language representation of new knowledge through graphic visualisations…”
BackgroundWang, D., Chen, Z., Florescu, I., & Wen, B. (2023) · Research in International Business and Finance · doi:10.1016/j.ribaf.2022.101869 · Cited by 36
Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI
Donoso-Guzmán, I., Ooge, J., Parra, D., & Verbert, K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_10 · Cited by 31
“– Novelty [8, 30, 32, 35, 43]: explanations should focus on abnormal causes [35] and provide information the user does not expect to increase her engagement with the system.”
BackgroundMethodInfluentialRule-Based Detection of Anomalous Patterns in Device Behavior for Explainable IoT Security
Costa, G., Forestiero, A., & Ortale, R. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3327822 · Cited by 24
“Explainable AI [11], [40], a sub-field of ML, aims to enhance the training, learning and representation processes providing Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundArtificial Intelligence for Improved Maternal Healthcare: A Systematic Literature Review
Chemisto, M., Gutu, T. J. L., Kalinaki, K., Mwebesa, D. B., Egau, P., Fred, K., Oloya, I. T., & KISITU, R. (2023) · IEEE AFRICON · doi:10.1109/africon55910.2023.10293674 · Cited by 27
“Popular metrics include predictive values, the area under the curve (AUC), receiver operating characteristic (ROC), sensitivity, probability ratios, and precision [12].”
BackgroundWang, Y., Gan, T., Zhao, N., Yin, G., Ye, Z., Sheng, R., Li, T., Liang, T., Renqing, 贾. J., Li, F., et al. (2024) · Spectrochimica Acta Part B Atomic Spectroscopy · doi:10.1016/j.sab.2024.107001 · Cited by 23
Qiu, J., Zhong, Y., Shao, Y., Zhang, G., Yang, J., Li, Z., & Cheng, Y. (2024) · Chemical Engineering Journal · doi:10.1016/j.cej.2024.155987 · Cited by 23
Bie, Y., Luo, L., Chen, Z., & Chen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_72 · Cited by 45
Porto, R., Molina, J. M., Berlanga, A., & Patricio, M. A. (2021) · Applied Sciences · doi:10.3390/app11031285 · Cited by 25
What Makes a Good Explanation?: A Harmonized View of Properties of ExplanationsPreprint
Chen, Z., Subhash, V., Havasi, M., Pan, W., & Doshi‐Velez, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.05667 · Cited by 37
“Carvalho et al. (2019) also classify explanation methods based on various criteria such as explanation type, result, scope, etc. Marcinkevics and Vogt (2020) provide a similar non-exhaustive categorization of various explainable and interpretable methods from a quantitative perspective.”
MethodMai, J., Wei, F., He, W., Huang, H., & Zhu, H. (2024) · Electronics · doi:10.3390/electronics13224358 · Cited by 24
Use of Explainable Artificial Intelligence for Analyzing and Explaining Intrusion Detection Systems
Hermosilla, P., Díaz, M., Berríos, S., & Allende, H. (2025) · Computers · doi:10.3390/computers14050160 · Cited by 22
Comprehensive and Dedicated Metrics for Evaluating AI-Generated Residential Floor Plans
Zeng, P., Yin, J., Gao, Y., Li, J., Jin, Z., & Lu, S. (2025) · Buildings · doi:10.3390/buildings15101674 · Cited by 22
Dai, Z., Zhan, C., Yin, H., Chen, J., Xu, L., Xia, Y., Yang, S., Chen, W., Cao, M., Du, Z., et al. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2025jh000703 · Cited by 50
Interpretable Machine Learning for Battery Prognosis: Retrospect and Prospect
Wang, T., Liu, K., Peng, H., & Liu, X. (2025) · Advanced Energy Materials · doi:10.1002/aenm.202503067 · Cited by 31
Systematic Literature Review on Explainable Learning Analytics and Educational Data Mining
Gunasekara, S., & Saarela, M. (2025) · IEEE Access · doi:10.1109/access.2025.3643645 · Cited by 25
“It is often associated with the clarity and distinctness of the boundaries between different groups in the feature space [182].”
Review article: Harnessing data-driven methods for climate multi-hazard and multi-risk assessment
Ferrario, D. M., Sanò, M., Maraschini, M., Critto, A., & Torresan, S. (2026) · Natural hazards and earth system sciences · doi:10.5194/nhess-26-2975-2026 · Cited by 25
XAI for Operations in the Process Industry - Applications, Theses, and Research Directions.
Kotriwala, A. M., Klöpper, B., Dix, M., Gopalakrishnan, G., Ziobro, D., & Potschka, A. (2021) · AAAI Spring Symposium Combining Machine Learning with Knowledge Engineering · Cited by 22
LoMEF: A framework to produce local explanations for global model time series forecasts
Rajapaksha, D., Bergmeir, C., & Hyndman, R. J. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.06.006 · Cited by 23
VisRuler: Visual analytics for extracting decision rules from bagged and boosted decision trees
Chatzimparmpas, A., Martins, R. M., & Kerren, A. (2023) · Information Visualization · doi:10.1177/14738716221142005 · Cited by 25
The non-linear nature of the cost of comprehensibility
Goethals, S., Martens, D., & Evgeniou, T. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00579-2 · Cited by 27
“The lack of comprehensibility in black box models is one of their main pitfalls, as their inner working is hidden to the users preventing them from verifying whether the reasoning of the system is, for example, aligned with restrictions or preferences of how decisions are made [19– 21].”
BackgroundElucidating precipitation in FeCrAl alloys through explainable AI: A case study
Ravi, S. K., Roy, I., Roychowdhury, S., Feng, B., Ghosh, S., Reynolds, C., Umretiya, R. V., Rebak, R. B., & Hoffman, A. (2023) · Computational Materials Science · doi:10.1016/j.commatsci.2023.112440 · Cited by 27
Designing and Evaluating User Experience of an AI-Based Defense System
Park, S., Kim, H. K., Park, J., & Lee, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3329257 · Cited by 25
Hu, B., Jiang, L., Zhang, S., & Wang, Q. (2023) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2023.3347278 · Cited by 23
“Moreover, it provides a high level of explainability [11].”
BackgroundContributions Estimation in Federated Learning: A Comprehensive Experimental Evaluation
Chen, Y., Li, K., Li, G., & Wang, Y. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3659437.3659459 · Cited by 22
“For example, in classification tasks, Macro F1 Score [8] is preferable over Accuracy when dealing with datasets that have extremely imbalanced classes.”
Evidence-based evaluation of a serious game to increase bullying awareness
Alonso‐Fernández, C., Calvo-Morata, A., Freire, M., Martínez‐Ortiz, I., & Fernández‐Manjón, B. (2020) · Interactive Learning Environments · doi:10.1080/10494820.2020.1799031 · Cited by 23
“This would not have been possible for other black-box models, which raises the widely-discussed need of explainable arti fi cial intelligence (xAI), as discussed in (Adadi & Berrada, 2018; Carvalho et al., 2019).”
BackgroundIntelligent knowledge consolidation: From data to wisdom
Hussain, M., Satti, F. A., Ali, S. I., Hussain, J., Ali, T., Kim, H., Yoon, K., Chung, T., & Lee, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107578 · Cited by 22
Korica-Pehserl, P., Gayar, N. E., & Pang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91608-4_33 · Cited by 24
“• Safety: We need to further research on security of explainability of AI, including methods to prohibit the fooling of XAI methods through perturbations and randomized input as well as methods to mitigate inferring private and sensitive information through explanations, see [9, 10, 13, 17, 19-21].”
BackgroundMethodInterpretability Needs a New Paradigm
Madsen, A., Lakkaraju, H., Reddy, S., & Chandar, S. (2024) · arXiv · doi:10.48550/arxiv.2405.05386 · Cited by 22
“Based on this and other sources (Madsen et al., 2022b; Arya et al., 2019; Carvalho et al., 2019; Murdoch et al., 2019), this paper refers to these two ideas as 1) the intrinsic paradigm and 2) the post-hoc paradigm.”
BackgroundMethodLong-Term Stability of Low-Cost IoT System for Monitoring Water Quality in Urban Rivers
Naloufi, M., Abreu, T. W. M., Souihi, S., Thérial, C., Rodrigues, N. A. D. P., Goff, A. G. –. L., Saad, M., Vinçon‐Leite, B., Dubois, P., Delarbre, M., et al. (2024) · Water · doi:10.3390/w16121708 · Cited by 21
“In addition, in situ sensing devices combined with machine learning could help stakeholders to detect in real time the possible contamination and to optimize the sampling effort [4,5].”
Pickering, L., Cohen, K. D. O., & Baets, B. D. (2025) · International Journal of Fuzzy Systems · doi:10.1007/s40815-025-02022-z · Cited by 37
“Carvalho et al. [27] prefer qualitative indicators.”
InfluentialExplainable AI Framework for Software Defect Prediction
Geçer, B. G., & Tarhan, A. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70018 · Cited by 33
Explainable AI (XAI) in Healthcare: Bridging the Gap between Accuracy and Interpretability
Ozdemir, O., & Fatunmbi, T. O. (2024) · Journal of Science Technology and Engineering Research · doi:10.64206/0z78ev10 · Cited by 28
Muralidhar, D., Belloum, R., & Ashok, A. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103591 · Cited by 25
Explainable artificial intelligence for molecular design in pharmaceutical research
Lamens, A., & Bajorath, J. (2026) · Chemical Science · doi:10.1039/d5sc08461j · Cited by 25
Evaluating Attribution Methods in Machine Learning Interpretability
Ratul, Q. E. A., Serra, E., & Cuzzocrea, A. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671501 · Cited by 30
“Hence many interpretable models and explanation methods [2] are developed.”
MethodChange Detection for Local Explainability in Evolving Data Streams
Haug, J., Braun, A., Zürn, S., & Kasneci, G. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557257 · Cited by 25
“For more detailed information, we refer to recent surveys [2, 4, 12, 22].”
A quantitative evaluation of explainable AI methods using the depth of decision tree
AHMED, N. A. M., & Alpkoçak, A. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3924 · Cited by 23
“As a result, the depth of the decision tree can indicate how complex the XAI method is [7, 8, 15].”
BackgroundMethodTest Input Prioritization for Machine Learning Classifiers
Dang, X., Li, Y., Papadakis, M., Klein, J., Bissyandé, T. F., & Traon, Y. L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3350019 · Cited by 29
“Machine learning testing involves systematically evaluating and validating machine learning models to ensure their accuracy, reliability, and effectiveness in prediction or decision-making [7], [51], [52], [53].”
BackgroundNoviandy, T. R., Zahriah, Z., Yandri, E., Jalil, Z., Yusuf, M. Z., Yusof, N. I. S. M., Lala, A., & Idroes, R. (2024) · Journal of Educational Management and Learning · doi:10.60084/jeml.v2i1.191 · Cited by 21
Exploring Interpretable AI Methods for ECG Data Classification
Ojha, J., Haugerud, H., Yazidi, A., & Lind, P. G. (2024) · doi:10.1145/3643488.3660294 · Cited by 22
“The high complexity of AI models - particularly those that follow the so-called deep learning (DL) paradigm [10]- obscures the rationale behind the decisions retrieved as an outcome of an algorithm [8].”
BackgroundZhang, C., Chen, Y., Wei, Y., Yu, P., Hong, Y., Hu, Y., Wang, J., & Shi, Z. (2024) · Geoderma · doi:10.1016/j.geoderma.2024.117032 · Cited by 21
Selection Problems in the Presence of Implicit Bias
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603209 · Cited by 22
Hessian-based toolbox for reliable and interpretable machine learning in physics
Dawid, A., Huembeli, P., Tomza, M., Lewenstein, M., & Dauphin, A. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac338d · Cited by 26
Statistical arbitrage powered by Explainable Artificial Intelligence
Carta, S., Consoli, S., Podda, A. S., Recupero, D. R., & Stanciu, M. M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117763 · Cited by 26
Assessment and Optimization of Explainable Machine Learning Models Applied to Transcriptomic Data
Zhao, Y., Shao, J., & Asmann, Y. W. (2022) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2022.07.003 · Cited by 25
“Typically, complicated models are not easily interpretable [36], which is also confirmed by the poor performance when interpreting CNN-based models.”
BackgroundI-SPLIT: Deep Network Interpretability for Split Computing
Cunico, F., Capogrosso, L., Setti, F., Damiano Carra, B., Fummi, F., & Cristani, M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956625 · Cited by 27
“It can be distinguished into intrinsic and post-hoc [29]: intrinsic interpretability applies to models that have a simple analytical form such as short decision trees or sparse linear models [30]; post-hoc interpretability”
BackgroundViswanathan, S., Ibrahim, S. B., Shankar, R., Binns, R., Kleek, M. G. V., & Slovák, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714088 · Cited by 26
“While XAI tools enable expert users to grasp the rationale behind AI outputs, [21, 40, 65], most non-experts do not engage with XAI [66], highlighting the need for innovative methods to foster AI understanding [66].”
Robustness and Cybersecurity in the EU Artificial Intelligence Act
Nolte, H., Rateike, M., & Finck, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732020 · Cited by 34
“In ML, the metric accuracy typically describes the overall proportion of correct predictions out of the total number of predictions made [12].”
Liu, P., Pan, D., Jiao, X.-Y., Liu, J., Du, P., Li, P.-C., Xue, M.-Z., Jin, Y., Wang, C.-S., Wang, X.-R., et al. (2025) · npj Clean Water · doi:10.1038/s41545-025-00504-z · Cited by 22
“These metrics are particularly valuable in environmental contexts where source labels may be subject to annotation errors 65 .”
The Externalities of Exploration and How Data Diversity Helps Exploitation
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603208 · Cited by 21
Rajapaksha, D., & Bergmeir, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21469 · Cited by 26
Explainability in Automatic Short Answer Grading
Schlippe, T., Stierstorfer, Q., ten Koppel, M., & Libbrecht, P. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-19-8040-4_5 · Cited by 20
Machine Learning Methods Improve Specificity in Newborn Screening for Isovaleric Aciduria
Zaunseder, E., Mütze, U., Garbade, S. F., Haupt, S., Feyh, P., Hoffmann, G. F., Heuveline, V., & Kölker, S. (2023) · Metabolites · doi:10.3390/metabo13020304 · Cited by 21
“The application of LR allows for an interpretation of its feature contributions, as it is interpretable on a modular level, meaning that it can be inherently explained how parts of the model affect predictions [32].”
BackgroundEfficient evolution of decision trees via fully matrix-based fitness evaluation
Costa, V. G., Salcedo‐Sanz, S., & Pedreira, C. E. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.111045 · Cited by 22
Explainable Machine Learning in Risk Management: Balancing Accuracy and Interpretability
Wang, M., Zhang, X., Yang, Y., & Wang, J. (2025) · Journal of Financial Risk Management · doi:10.4236/jfrm.2025.143011 · Cited by 22
Understanding the Prediction Mechanism of Sentiments by XAI Visualization
So, C. (2020) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3443279.3443284 · Cited by 22
Diagnosing AI Explanation Methods with Folk Concepts of Behavior
Jacovi, A., Bastings, J., Gehrmann, S., Goldberg, Y., & Filippova, K. (2023) · Journal of Artificial Intelligence Research · doi:10.1145/3593013.3593993 · Cited by 30
Machine Learning post-hoc interpretability: a systematic mapping study
Vieira, C. P. R., & Digiampietri, L. A. (2022) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3535511.3535512 · Cited by 25
“The classification of existing work for interpretable methods mainly focuses on two categories: intrinsic interpretability and post-hoc interpretability [6, 8].”
MethodExplainable AI applications in the Medical Domain: a systematic review
Prentzas, N., Kakas, A., & Pattichis, C. S. (2023) · arXiv · doi:10.48550/arxiv.2308.05411 · Cited by 19
Stephen, C., Basu, B., & McNabola, A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129167 · Cited by 19
Comparing Strategies for Post-Hoc Explanations in Machine Learning Models
Vij, A., & Nanjundan, P. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-1866-6_41 · Cited by 19
Benjamin, J. J., Kinkeldey, C., Müller-Birn, C., Korjakow, T., & Herbst, E.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492858 · Cited by 19
“However, studies of interpretability predominantly consider only an ML expert or data scientist perspective, are often model-centric, or provide a mathematical assessment of the model results [1, 11].”
BackgroundMethodEldrandaly, K. A., Abdel‐Basset, M., Ibrahim, M., & AbdelAziz, N. M. (2022) · Enterprise Information Systems · doi:10.1080/17517575.2022.2098537 · Cited by 21
“It ensures that small differences in the input do not have a major impact on the forecast (Carvalho, Pereira, and Cardoso 2019).”
BackgroundMethodInfluentialAI-Based Exhaust Gas Temperature Prediction for Trustworthy Safety-Critical Applications
Apostolidis, A., Bouriquet, N., & Stamoulis, K. (2022) · Aerospace · doi:10.3390/aerospace9110722 · Cited by 19
“Further information about the interpretability of machine learning models is available in [22] as well as in [23].”
BackgroundThe Impact of Information Technology Integration on the Decision-Making Process
Hussein, R. N., Nassreddine, G., & Younis, J. A. (2023) · Journal of Techniques · doi:10.51173/jt.v5i1.1262 · Cited by 33
Explainable Artificial Intelligence for Patient Safety: A Review of Application in Pharmacovigilance
Lee, S., Kim, S., Lee, J., Kim, J., Song, M.-H., & Lee, S. (2023) · IEEE Access · doi:10.1109/access.2023.3271635 · Cited by 21
“Knowledge extraction is a form of model-specific XAI that extracts knowledge from a large, complex model to a more simplified model, and for many years model compression [60], tree regularization [61], and a combination of model compression and dimensionality reduction [62].”
BackgroundAn interpretable machine learning framework for diagnosis and prognosis of COVID-19
Fan, Y., Liu, M., & Sun, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0291961 · Cited by 19
Kundu, R. K., & Hoque, K. A. (2023) · Annual Conference of the PHM Society · doi:10.36001/phmconf.2023.v15i1.3472 · Cited by 19
“Note, an explanation with a low fidelity score can be useless (Carvalho, Pereira, & Cardoso, 2019).”
BackgroundGrigoraș, C. C., Zichil, V., Ciubotariu, V. A., & Cosa, S. M. (2024) · Machines · doi:10.3390/machines12030180 · Cited by 19
Improving the maize crop row navigation line recognition method of YOLOX
Gong, H., Zhuang, W., & Wang, X. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1338228 · Cited by 19
Zhang, C., Hofmann, F., Plößl, L., & Gläser‐Zikuda, M. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12720-0 · Cited by 23
“For example, interpretability in model decision-making refers to the transparency of how a model processes inputs to produce outputs (e.g., Carvalho et al., 2019).”
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Monea, G., Peyrard, M., Josifoski, M., Chaudhary, V., Eisner, J. M., Kıcıman, E., Palangi, H., Patra, B., & West, R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.369 · Cited by 32
Kordbagheri, A., Kordbagheri, M., Tayim, N., Fakhrou, A. A., & Davoudi, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109372 · Cited by 19
Tiamiyu, D., Aremu, S. O., Emmanuel, I., Ihejirika, C. J., Adewoye, M. B., & Ajayi, A. A. (2024) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst24116170 · Cited by 21
Deep Learning for Automatic Detection of Volcanic and Earthquake-Related InSAR Deformation
Liu, X., Zhang, Y., Shan, X., Wang, Z., Gong, W., & Zhang, G. (2025) · Remote Sensing · doi:10.3390/rs17040686 · Cited by 21
Calibrated explanations for regression
Löfström, T., Löfström, H., Johansson, U., Sönströd, C., & Matela, R. (2025) · Machine Learning · doi:10.1007/s10994-024-06642-8 · Cited by 22
“Stability refers to the consistency of the explanations [11, 14]; the same instance and model should produce identical explanations across multiple runs.”
Methodology for Interpretable Reinforcement Learning Model for HVAC Energy Control
Kotevska, O., Munk, P. L., Kurte, K. R., Du, Y., Amasyali, K., Smith, R. W., & Zandi, H. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9377735 · Cited by 18
“Different aspects contribute to defining the trust and in this work, we are exploring the interpretability aspect of contribution to increased trust in ML systems [7].”
BackgroundAn Efficient Iterative Approach to Explainable Feature Learning
Vlahek, D., & Mongus, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107049 · Cited by 21
“Nevertheless, due to their simplicity, these types of models do not represent sufficiently all the knowledge contained within the data [87] and, thus, often achieve low classification accuracies [88].”
BackgroundAUTOMOTIVE: A Case Study on AUTOmatic multiMOdal Drowsiness detecTIon for smart VEhicles
Esteves, T., Pinto, J. R., Ferreira, P. M., Costa, P. A., Rodrigues, L., Antunes, I., Lopes, G., Gamito, P. S., Abrantes, A. J., Jorge, P., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3128016 · Cited by 22
“The main contribution of this work for the AUTOMOTIVE project can be summarized as follows: • Identification of desirable properties for generalization of deep neural models to unseen data and attacks : the AUTOMOTIVE project moves forward by using interpretability tools to state the following…”
BackgroundSelf-citationMachine Reasoning ExplainabilityPreprint
Čyras, K., Badrinath, R., Mohalik, S. K., Mujumdar, A. P., Nikou, A., Previti, A., Sundararajan, V., & Feljan, A. V. (2020) · arXiv (Cornell University) · doi:10.4230/lipics.cp.2026.62 · Cited by 23
Interpretable Uncertainty Quantification in AI for HEP
Chen, T. Y., Dey, B., Ghosh, A., Kagan, M., Nord, B., & Ramachandra, N. (2022) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1886020 · Cited by 26
Prada, E. J. A., & García, J. F. C. (2024) · Sensors · doi:10.3390/s24227193 · Cited by 21
Polo, E. M., Iacomi, F., Rey, A. V., Ferraris, D., Paglialonga, A., & Barbieri, R. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110310 · Cited by 26
Tourab, H., Lopez-Perez, L., Arroyo, P., Georga, E. I., Rujas, M., Ponziani, F. R., Torrego-Ellacuría, M., Barbancho, B. M., Dei, N. N., Ciuti, G., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3593198 · Cited by 21
“We referred to the taxonomy of XAI methods from previously published papers [24], [25].”
Suffian, M., Kuhl, U., Bogliolo, A., & Alonso-Moral, J. M. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103484 · Cited by 19
Kares, F., Speith, T., Zhang, H., & Langer, M. (2026) · Computers in Human Behavior · doi:10.1016/j.chb.2026.109163 · Cited by 21
Parra, R., Ojeda, V., Noguera, J. L. V., García-Torres, M., Mello-Román, J. C., Villalba, C., Facon, J., Divina, F., Cardozo, O., Castillo, V. E., et al. (2021) · Diagnostics · doi:10.3390/diagnostics11111951 · Cited by 19
“[25] defines trust based on the answer to two questions: (1) How much trust do we have in a model that gives wrong answers with great confidence? and (2) How much trust do we have in a model that gives right answers hesitantly? However, valuable, interpretability and trust are known to be domain-specific notions [26].”
BackgroundComprehensible counterfactual explanation on Kolmogorov-Smirnov test
Cong, Z., Chu, L., Yu, Y., & Pei, J. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3461535.3461546 · Cited by 19
“A desirable idea is to find a counterfactual explanation that is most consistent with a user’s domain knowledge so that the explanation is best comprehensible to the user [17, 42].”
BackgroundMethodHuman-centric and semantics-based explainable event detection: a survey
Kolajo, T., & Daramola, O. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10525-0 · Cited by 22
“As a result, there is no universally accepted method for evaluating human-centred explanations (Li et al. 2022 ) because of the subjectivity of the reasonableness idea, insight, and interest of clients (Carvalho et al. 2019 ).”
BackgroundMethodComputational complexity in explainable decision support system: A review
Ezeji, I. N., Adigun, M. O., & Oki, O. A. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-219407 · Cited by 22
Ghosh, J., Choudhury, S. R., Singh, K., & Koner, S. (2024) · International Journal of Advancement in Life Sciences Research · doi:10.31632/ijalsr.2024.v07i02.004 · Cited by 19
Integrating Fuzzy C-Means Clustering and Explainable AI for Robust Galaxy Classification
Díaz, G. M., Medina, R. G., & Jiménez, J. A. A. (2024) · Mathematics · doi:10.3390/math12182797 · Cited by 19
“Inherently Interpretable Models [31] are designed to be simple and transparent, making their decision-making process easy to understand.”
Warnett, S. J., Ntentos, E., & Zdun, U. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112257 · Cited by 18
“A literature review conducted by Carvalho et al. [40] identifies established methods and metrics for interpreting ML models.”
Krishnamoorthy, U., & Balasubramani, S. (2024) · Advanced Theory and Simulations · doi:10.1002/adts.202400479 · Cited by 22
Popović, S., Viduka, D., Bašić, A., Dimić, V., Djukic, D., Nikolić, V. R., & Stokić, A. (2025) · Electronics · doi:10.3390/electronics14030562 · Cited by 20
Architecture Disentanglement for Deep Neural Networks
Hu, J., Cao, L., Tong, T., Ye, Q., Zhang, S., Li, K., Huang, F., Shao, L., & Ji, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00071 · Cited by 27
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.