Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 201–300 of 1,007
Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare
Procter, R., Tolmie, P., & Rouncefield, M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3577009 · Cited by 92
“As a consequence, much effort is now being devoted to developing ways of (cid:1744)opening up(cid:1745) the black box (Carvalho et al. 2019, Du et al. 2019, Meteier et al. 2019, Mittelstadt et al. 2019) through the development of techniques for explainable AI (xAI) (Adadi et al. 2018, Gilpin et al.…”
Leveraging explanations in interactive machine learning: An overview
Teso, S., Alkan, Ö., Stammer, W., & Daly, E. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1066049 · Cited by 106
“The main source of inspiration is the vast body of knowledge on explainable AI, which has been summarized in several overviews and surveys (Adadi and Berrada, 2018; Gilpin et al., 2018; Guidotti et al., 2018b; Montavon et al., 2018; Carvalho et al., 2019; Sokol and Flach, 2020a; Belle and Papantonis, 2021; Liao and Va…”
BackgroundExplainable Artificial Intelligence for Sarcasm Detection in Dialogues
Kumar, A., Dikshit, S., & de Albuquerque, V. H. C. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/2939334 · Cited by 86
“Two widely accepted categories of post hoc approaches are surrogate models and counterfactual explanations [14].”
BackgroundMethodInfluentialBarraza, J. E. F., Droguett, E. L., & Martins, M. R. (2021) · Sensors · doi:10.3390/s21175888 · Cited by 86
“According to the authors in [38], model interpretability frameworks can be divided into local and global interpretability.”
BackgroundEthically Responsible Machine Learning in Fintech
Rizinski, M., Peshov, H., Mishev, K., Chitkushev, L. T., Vodenska, I., & Trajanov, D. (2022) · IEEE Access · doi:10.1109/access.2022.3202889 · Cited by 89
“area with the goal of increasing model transparency [38], 848 [101], [102].”
BackgroundSelective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 53
Nikiforidis, K., Kyrtsoglou, A., Vafeiadis, T., Kotsiopoulos, T., Nizamis, A., Ioannidis, D., Votis, K., Tzovaras, D., & Sarigiannidis, P. G. (2024) · ICT Express · doi:10.1016/j.icte.2024.12.001 · Cited by 88
Dawar, I., Srivastava, A., Singal, M., Dhyani, N., & Rastogi, S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11196-9 · Cited by 86
A Review of Driving Style Recognition Methods From Short-Term and Long-Term Perspectives
Chu, H., Zhuang, H., Wang, W., Na, X., Guo, L., Zhang, J., Gao, B., & Chen, H. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3279425 · Cited by 82
“However, given the lack of uniformity in the definition of interpretability, it is still difficult to evaluate model interpretability [107].”
BackgroundBlanco, K., Salcidua, S., Orellana, P., Sauma-Pérez, T., León, T., Steinmetz, L. C. L., Ibañez, A., Duran-Aniotz, C., & de la Cruz, R. (2023) · Alzheimer's Research & Therapy · doi:10.1186/s13195-023-01304-8 · Cited by 73
“This occurs mainly in modeling approaches based on neural networks (one of the most popular in use) [77].”
Gawde, S., Patil, S., Kumar, S., Kamat, P. V., & Kotecha, K. V. (2024) · Decision Analytics Journal · doi:10.1016/j.dajour.2024.100425 · Cited by 73
Machine learning prediction of enzyme optimum pH
Gado, J. E., Knotts, M., Shaw, A. Y., Marks, D. S., Gauthier, N. P., Sander, C., & Beckham, G. T. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01026-6 · Cited by 87
“While several interpretability methods are available 63 , in this work, we assessed the attention weights per residue in the RLATtr top model (output of the softmax layer) to identify the residues EpHod emphasizes for inference (Fig.”
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
Dai, J. C., Upadhyay, S., Aïvodji, U., Bach, S. H., & Lakkaraju, H. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534159 · Cited by 127
“[80], who propose two high-level goals of explanation methods: interpretability (the clarity, simplicity, and broadness of the explanations), and fidelity (the completeness and soundness of explanations) [19, 32].”
BackgroundTe’eni, D., Yahav, I., Zagalsky, A., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. (2023) · Management Science · doi:10.1287/mnsc.2022.03518 · Cited by 91
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71
“Such high-dimensional effects can cause instabilities as illustrated, for example, by the emergence of so-called adversarial examples, see for example, References [22,23].”
BackgroundInfluentialClinical data classification with noisy intermediate scale quantum computers
Moradi, S., Brandner, C., Spielvogel, C. P., Krajnc, D., Hillmich, S., Wille, R., Drexler, W., & Papp, L. (2022) · Scientific Reports · doi:10.1038/s41598-022-05971-9 · Cited by 81
Choraś, M., Pawlicki, M., Puchalski, D., & Kozik, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50423-6_46 · Cited by 86
“Similar categorization is given in [38] where in-model (integrated/intrinsic) and post-model (post-hoc) methods exist alongside additional pre-model inter-pretability methods.”
BackgroundMethodInfluentialDivine, D. C., Hubert, S., Epelle, E. I., Ojo, A. U., Adeleke, A. A., Ogbaga, C. C., Akande, O., Okoye, P. U., Giwa, A., & Okolie, J. A. (2024) · Fuel · doi:10.1016/j.fuel.2024.131346 · Cited by 82
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
An Interpretable Machine Learning Model for Daily Global Solar Radiation Prediction
Chaibi, M., Benghoulam, E. M., Tarik, L., Berrada, M., & Hmaidi, A. E. (2021) · Energies · doi:10.3390/en14217367 · Cited by 71
“Model-agnostic can be applied to any ML algorithm (e.g., PFI, PDP, SHAP), while model-specific is limited to specific model classes [37].”
BackgroundMethodSakai, A., Komatsu, M., Komatsu, R., Matsuoka, R., Yasutomi, S., Dozen, A., Shozu, K., Arakaki, T., Machino, H., Asada, K., et al. (2022) · Biomedicines · doi:10.3390/biomedicines10030551 · Cited by 68
“One of the major obstacles is regarded as the “black box problem” of AI [4,6–8].”
BackgroundExplaining sentiment analysis results on social media texts through visualization
Jain, R., Kumar, A., Nayyar, A., Dewan, K., Garg, R., Raman, S., & Ganguly, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14432-y · Cited by 68
“Interpretability is a concept that contributes to the development of responsible AI and can be achieved on a task [12, 32, 41].”
BackgroundPoisoning attacks and countermeasures in intelligent networks: Status quo and prospects
Wang, C., Chen, J., Yang, Y., Ma, X., & Liu, J. (2021) · Digital Communications and Networks · doi:10.1016/j.dcan.2021.07.009 · Cited by 90
Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603203 · Cited by 65
Achieving Explainability of Intrusion Detection System by Hybrid Oracle-Explainer Approach
Szczepanski, M., Choras, M., Pawlicki, M., & Kozik, R. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207199 · Cited by 64
Some critical and ethical perspectives on the empirical turn of AI interpretability
John-Mathews, J.-M. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121209 · Cited by 86
Tamagusko, T., & Ferreira, A. (2023) · Infrastructures · doi:10.3390/infrastructures8120170 · Cited by 72
“Additionally, the lack of interpretability limits the transparency of the ML models, making it difficult to assess their reliability and validity [81–83].”
BackgroundGongane, V. U., Munot, M. V., & Anuse, A. D. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00248-9 · Cited by 90
“In Model Induction XAI method, model-agnostic method provides interpretations after the ML model is trained and are independent of ML model [24, 52].”
InfluentialThe Explainability of Transformers: Current Status and Directions
Fantozzi, P., & Naldi, M. (2024) · Computers · doi:10.3390/computers13040092 · Cited by 83
An Empirical Evaluation of AI Deep Explainable Tools
Hailemariam, Y., Yazdinejad, A., Parizi, R. M., Srivastava, G., & Dehghantanha, A. (2020) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps50303.2020.9367541 · Cited by 90
“Diogo et al. [26] proposed that explanation quality can be measured by correctness.”
BackgroundPhilosophy of science at sea: Clarifying the interpretability of machine learning
Beisbart, C., & Räz, T. (2022) · Philosophy Compass · doi:10.1111/phc3.12830 · Cited by 58
“Interpretability has thus become an important research topic in computer science (e.g., Carvalho et al., 2019; Chakraborty et al., 2017; Reyes et al., 2020).”
BackgroundBridging the Transparency Gap: What Can Explainable AI Learn from the AI Act?
Gyevnár, B., Ferguson, N., & Schäfer, B. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230367 · Cited by 79
Explainable Artificial Intelligence Based Framework for Non-Communicable Diseases Prediction
Davagdorj, K., Bae, J.-W., Pham, V.-H., Theera-Umpon, N., & Ryu, K. H. (2021) · IEEE Access · doi:10.1109/access.2021.3110336 · Cited by 60
“In general, accurate prediction performance and explainability are two dominant criteria of the best decision support system [14].”
BackgroundRecent Methods for Evaluating Crop Water Stress Using AI Techniques: A Review
Cho, S. B., Soleh, H. M., Choi, J. W., Hwang, W.-H., Lee, H., Cho, Y.-S., Cho, B., Kim, M. S., Baek, I., & Kim, G. (2024) · Sensors · doi:10.3390/s24196313 · Cited by 74
“However, developing technologies that can adapt to accelerating climate change and variable production conditions remains a crucial challenge.”
BackgroundExplainable AI for industrial fault diagnosis: A systematic review
Cação, J., Santos, J., & Antunes, M. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100905 · Cited by 86
Applications of physics informed neural operators
Rosofsky, S., Majed, H. A., & Huerta, E. A. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acd168 · Cited by 73
“We will also continue our research program combining scien-tific visualization and accelerated computing to gain insights about what PINOs learn from data, and how this information may be used to enhance their performance when applied to experimental datasets [41–43].”
MethodAcademic achievement prediction in higher education through interpretable modeling
Wang, S., & Luo, B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309838 · Cited by 86
“Broadly, interpretable machine learning methods are divided into two categories: self-inter-pretation models and post-hoc interpretation methods [34].”
MethodMachine Learning and Prediction in Psychological Assessment: Some Promises and Pitfalls
Fokkema, M., Iliescu, D., Greiff, S., & Ziegler, M. D. (2022) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000714 · Cited by 66
Lavecchia, A. (2025) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.70049 · Cited by 86
“Standard evaluation metrics include fidelity, interpretability, stability, completeness, and sparsity [89].”
Using Kernel SHAP XAI Method to Optimize the Network Anomaly Detection Model
Roshan, K., & Zafar, A. (2022) · International Conference on Computing for Sustainable Global Development · doi:10.23919/indiacom54597.2022.9763241 · Cited by 75
“DL based models) over the years [4] [5].”
MethodPei, T., & Qiu, T. (2024) · Acta Geotechnica · doi:10.1007/s11440-024-02384-y · Cited by 63
“…gained traction in various fields, including computer vision and natural language processing, and some regulated domains that require high-stakes predictions, such as healthcare, criminal justice, and finance, as detailed in comprehensive surveys by Carvalho et al. [15] and von Rueden et al. [88].”
On the Relation of Trust and Explainability: Why to Engineer for Trustworthiness
Kastner, L., Langer, M., Lazar, V., Schomacker, A., Speith, T., & Sterz, S. (2021) · 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00031 · Cited by 61
Comparison of XGBoost and the Neural Network model on the class-balanced datasets
Wu, J., Li, Y., & Ma, Y. (2021) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic54370.2021.9647373 · Cited by 61
Ramesh, D., Kameswaran, V., Wang, D., & Sambasivan, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533237 · Cited by 47
“, of source-code, datasets, models, and processes surrounding the development of ADS) [21, 30, 55, 64, 83, 105] and explainable decisions (i.”
BackgroundA new approach based on association rules to add explainability to time series forecasting models
Troncoso-García, A. R., Martínez-Ballesteros, M., Alvarez, F. M., & Troncoso, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.021 · Cited by 68
“In the major cases, explainable models are applied to essential areas, such as disease detection in health care [11].”
BackgroundMethodCorn leaf disease: insightful diagnosis using VGG16 empowered by explainable AI
Tariq, M., Ali, U., Abbas, S., Hassan, S. G., Naqvi, R. A., Khan, M. A., & Jeong, D. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1402835 · Cited by 68
“While the de fi nition of interpretability differs from explainability in machine learning, both terms denote more or less the same meaning (Carvalho et al., 2019).”
BackgroundInterpretability of deep neural networks: A review of methods, classification and hardware
Antamis, T., Drosou, A., Vafeiadis, T., Nizamis, A., Ioannidis, D., & Tzovaras, D. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128204 · Cited by 70
Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach
Kolyshkina, I., & Simoff, S. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.660206 · Cited by 60
Development of an Explainable Prediction Model of Heart Failure Survival by Using Ensemble Trees
Moreno-Sánchez, P. A. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378460 · Cited by 67
“In the healthcare domain, lack of explanability limits a wider adoption of ML solutions since healthcare workers often find it challenging to trust complex ML models since they require a high technical and statistics knowledge [15].”
BackgroundExplainable Artificial Intelligence for Smart City Application: A Secure and Trusted Platform
Kabir, M. H., Hasan, K. F., Hasan, M. K., & Ansari, K. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-96630-0_11 · Cited by 72
Balloccu, G., Boratto, L., Fenu, G., & Marras, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532041 · Cited by 60
“Their feature relevance scores are a source of model interpretability rather than explainability from the user’s perspective [7].”
BackgroundRobustness, Stability, and Fidelity of Explanations for a Deep Skin Cancer Classification Model
Saarela, M., & Geogieva, L. (2022) · Applied Sciences · doi:10.3390/app12199545 · Cited by 66
“As pointed out by Carvalho et al. [38], an explanation with low fidelity is essentially useless.”
BackgroundThe Impacts of Open Data and eXplainable AI on Real Estate Price Predictions in Smart Cities
Neves, F. T., Aparício, M., & Neto, M. D. C. (2024) · Applied Sciences · doi:10.3390/app14052209 · Cited by 74
“Their opaqueness poses challenges as users, regardless of expertise, cannot fully understand or verify the rationale behind the system’s decisions [86].”
BackgroundEstablishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59
From human-centered to symbiotic artificial intelligence: a focus on medical applications
Desolda, G., Esposito, A., Lanzilotti, R., Piccinno, A., & Costabile, M. F. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20414-5 · Cited by 69
An Interpretable Artificial Intelligence Based Smart Agriculture System
Sabrina, F., Sohail, S., Farid, F., Jahan, S., Ahamed, F., & Gordon, S. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.026363 · Cited by 58
ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective
Huang, J., Mishra, A., Kwon, B. C., & Bryan, C. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209384 · Cited by 47
“Desiderata for this type of explanation includes faithfulness, consistency with prior beliefs and generality [4].”
BackgroundParakh, A., Awate, A., Barman, S. M., Kadu, R. K., Tulaskar, D. P., Kulkarni, M. B., & Bhaiyya, M. L. (2025) · Trends in Environmental Analytical Chemistry · doi:10.1016/j.teac.2025.e00280 · Cited by 63
Combining deep learning with token selection for patient phenotyping from electronic health records
Yang, Z., Dehmer, M., Yli-Harja, O. P., & Emmert‐Streib, F. (2020) · Scientific Reports · doi:10.1038/s41598-020-58178-1 · Cited by 65
Opportunities for Explainable Artificial Intelligence in Aerospace Predictive Maintenance
Shukla, B., Fan, I., & Jennions, I. K. (2020) · PHM Society European Conference · doi:10.36001/phme.2020.v5i1.1231 · Cited by 71
“The model-independent XAI models may apply to any model, but they may be more limited compared to the model-specific models (Carvalho, 2019).”
MethodHuman-Swarm-Teaming Transparency and Trust Architecture
Hepworth, A. J., Baxter, D. P., Hussein, A., Yaxley, K. J., Debie, E., & Abbass, H. A. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003545 · Cited by 50
“[105] further developed from Swartout and Moore’s description [50], can be directly applied to HST through Lyons’s four transparency models [66].”
MethodHow Much Automation Does a Data Scientist Want?Preprint
Wang, D., Liao, Q. V., Zhang, Y., Khurana, U., Samulowitz, H., Park, S., Muller, M. J., & Amini, L. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03970 · Cited by 66
“A review of the technical field of XAI can be found in recent survey papers [2, 8, 20, 37, 42].”
BackgroundTowards Explainability for AI Fairness
Zhou, J., Chen, F., & Holzinger, A. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_18 · Cited by 43
Belfield, S. J., Cronin, M., Enoch, S. J., & Firman, J. W. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282924 · Cited by 61
“Indeed, many such techniques produce algorithms which are in effect a “black box”–their inner workings hidden to the user, and the reasoning underlying their predictions largely opaque [69].”
BackgroundHow well can a large language model explain business processes as perceived by users?
Fahland, D., Fournier, F., Limonad, L., Skarbovsky, I., & Swevels, A. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102416 · Cited by 61
Byeon, H. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18147625 · Cited by 61
“The first goal of this approach was to compare the predictive performance (accuracy) of the single model (base model), because previous studies [13,17,18,21,22,33], which tried to predict diseases using single machine learning, commonly used them and reported them as highly-accurate models.”
MethodResultBanovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55
“However, most existing explainable AI (XAI) methods [10, 21, 43] target math-savy AI creators [1, 22] to aid them in AI model debugging and monitoring [6].”
BackgroundCondition-based maintenance using machine learning and role of interpretability: a review
Sharma, J., Mittal, M. L., & Soni, G. (2022) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-022-01843-7 · Cited by 75
“PdM results in reduced maintenance costs compared to CBM (Carvalho et al. 2019a, b; Ran et al. 2019).”
Liu, W., Kerekes, T., Dragičević, T., & Teodorescu, R. (2023) · IEEE Journal of Emerging and Selected Topics in Industrial Electronics · doi:10.1109/jestie.2023.3236885 · Cited by 57
“In addition, the lack of satisfactory interpretability has slowed down the progress of deep neural networks for engineering applications as an open problem due to their complex internal relationships [67], but this does not prevent AI from excelling in grid stability assessment (GSA).”
BackgroundInterpretable and explainable predictive machine learning models for data-driven protein engineering
Medina-Ortiz, D., Khalifeh, A., Anvari-Kazemabad, H., & Davari, M. D. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108495 · Cited by 63
Neto, E. C. P., Baum, D. M., de Almeida, J. R., Camargo, J. B., & Cugnasca, P. S. (2023) · Aerospace · doi:10.3390/aerospace10040358 · Cited by 54
Roussel, C., & Böhm, K. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12090355 · Cited by 60
Optimization Schedule Schemes for Charging Electric Vehicles: Overview, Challenges, and Solutions
Al-Alwash, H., Borcoci, E., Vochin, M., Balapuwaduge, I. A. M., & Li, F. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3371890 · Cited by 54
“In [69] and [70], an overview and a taxonomy of inter-pretable models and explanation methods based on different criteria were presented.”
BackgroundMethodExplainable AI in Education: Techniques and Qualitative Assessment
Gunasekara, S., & Saarela, M. (2025) · Applied Sciences · doi:10.3390/app15031239 · Cited by 53
“In order to improve explainability and make sure that forecasts are transparent, comprehensible, and reliable, the majority of the work in the field of ML explanation research is concentrated on developing new methods and approaches [3].”
InfluentialShobayo, O., & Saatchi, R. (2025) · Diagnostics · doi:10.3390/diagnostics15091072 · Cited by 53
XAI-AV: Explainable Artificial Intelligence for Trust Management in Autonomous Vehicles
Mankodiya, H., Obaidat, M. S., Gupta, R., & Tanwar, S. (2021) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci52664.2021.9583190 · Cited by 53
Explainable inflation forecasts by machine learning models
Aras, S., & Lisboa, P. J. G. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117982 · Cited by 62
Mill, E., Garn, W., Ryman-Tubb, N. F., & Turner, C. J. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01405121 · Cited by 64
Valente, F., Paredes, S., Henriques, J. M. O., Rocha, T., Carvalho, P., & Morais, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00821-8 · Cited by 62
Explainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 61
“Furthermore, explainability can support the reliability of a system [70].”
BackgroundMachine learning robustness: a primer
Braiek, H. B., & Khomh, F. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00012-2 · Cited by 78
Montenegro, H., Silva, W., & Cardoso, J. S. (2021) · IEEE Access · doi:10.1109/access.2021.3124844 · Cited by 59
“To overcome this problem, several interpretability methods have been proposed to enhance the transparency in the decisionmaking process and improve the trust in the results of the models [5].”
MethodSelf-citationA collective AI via lifelong learning and sharing at the edge
Soltoggio, A., Ben-Iwhiwhu, E., Braverman, V., Eaton, E., Epstein, B., Ge, Y., Halperin, L., How, J., Itti, L., Jacobs, M. A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00800-2 · Cited by 51
“In addition to established ML metrics 91 and recently developed LL metrics 36,37,45–47, 92 , ShELL-specific ShELL-specific Considering a single LL agent as a baseline, two metrics can be defined to measure objective 1: (1) how much performance gains can n agents achieve within a determined time…”
Anomaly diagnosis of connected autonomous vehicles: A survey
Fang, Y., Min, H., Wu, X., Wang, W., Zhao, X., Martínez‐Pastor, B., & Teixeira, R. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102223 · Cited by 57
Asset Management in Machine Learning: State-of-research and State-of-practice
Idowu, S., Strüber, D., & Berger, T. (2022) · ACM Computing Surveys · doi:10.1145/3543847 · Cited by 52
“, model explainability [16, 67] and interpretability [17, 31]) should be considered in future tools for improved result interpretation and decision making.”
BackgroundMethodDavazdahemami, B., Zolbanin, H. M., & Delen, D. (2022) · Decision Support Systems · doi:10.1016/j.dss.2022.113730 · Cited by 50
“3 See [89,90] for a comprehensive review of such methods.”
Nakanishi, T. (2023) · IEEE Access · doi:10.1109/access.2023.3314336 · Cited by 56
He, X., Hong, Y., Zheng, X., & Zhang, Y. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2095093 · Cited by 43
“In terms of coun-terfactual explanation, the academic community generally considers it important (Carvalho et al., 2019; Liao et al., 2020; Miller, 2019; Wang et al., 2019).”
BackgroundC-SHAP: A Hybrid Method for Fast and Efficient Interpretability
Ranjbaran, G., Recupero, D. R., Roy, C. K., & Schneider, K. A. (2025) · Applied Sciences · doi:10.3390/app15020672 · Cited by 54
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603207 · Cited by 45
Bianchi, M. G., Vitor, W. G., de Paiva, V. F., Okita, A. L., de Sousa, R. M., & Machado, B. S. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1241484 · Cited by 60
“This effect is called confirmation bias (25) and that is why our dermatologists assessed the dermoscopic images and Grad-CAM visual maps separately and blindly, trying to avoid it.”
BackgroundLiu, M., Wen, Z., & Su, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122022 · Cited by 51
Xu, M. R., Mei, Y., Zhang, F., & Zhang, M. (2024) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2024.3363970 · Cited by 48
“The interpretability of a model affects people ’ s trust in its real-world applications, especially for problems with serious ethical concerns [50].”
BackgroundWu, Y., Lin, S., Shi, K., Ye, Z., & Fang, Y. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-18913-9 · Cited by 50
“Thus, the demand for interpretable ML systems has also increased (Carvalho et al. 2019).”
BackgroundSanchis‐Segura, C., Aguirre, N., Cruz-Gómez, Á. J., Félix, S., & Forn, C. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119343 · Cited by 57
Opening the black box: interpretable machine learning for predictor finding of metabolic syndrome
Zhang, Y., Zhang, X., Razbek, J., Li, D., Xia, W., Bao, L., Mao, H., Daken, M., & Cao, M. (2022) · BMC Endocrine Disorders · doi:10.1186/s12902-022-01121-4 · Cited by 49
“Different classifications of interpretation methods can be made based on different criteria, grouping them according to when they are applied: before, during, and after building a machine learning model [3].”
Maloney, K. O., Buchanan, C., Jepsen, R. D., Krause, K. P., Cashman, M. J., Gressler, B. P., Young, J. A., & Schmid, M. C. (2022) · Journal of Environmental Management · doi:10.1016/j.jenvman.2022.116068 · Cited by 49
The Interacting Multiple Model Smooth Variable Structure Filter for Trajectory Prediction
Akhtar, S., & Habibi, S. R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3271295 · Cited by 47
“At this point, research is being done in the area of interpretable artificial intelligence (AI) to understand how neural networks function [43].”
BackgroundTowards data and analytics driven B2B-banking for green finance: A cross-selling use case study
Chang, V. I. C., Hahm, N., Xu, Q. A., Vijayakumar, P., & Liu, L. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2024.123542 · Cited by 48
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.