Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 101–200 of 1,007
Interpretable Machine Learning: A brief survey from the predictive maintenance perspective
Vollert, S., Atzmueller, M., & Theissler, A. (2021) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa45728.2021.9613467 · Cited by 174
Yang, L., Driscol, J., Sarigai, S., Wu, Q., Lippitt, C. D., & Morgan, M. (2022) · Sensors · doi:10.3390/s22062416 · Cited by 159
“The lack of interpretability raises major concerns across several domains; for example, in high-stakes prediction applications, such as autonomous driving, healthcare, and financial services [94], the trust of DL models is critical.”
BackgroundExploring Convolutional Neural Network Architectures for EEG Feature Extraction
Rakhmatulin, I., Dao, M.-S., Nassibi, A., & Mandic, D. P. (2024) · Sensors · doi:10.3390/s24030877 · Cited by 162
Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue
Chazette, L., Brunotte, W., & Speith, T. (2021) · 2021 IEEE 29th International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00025 · Cited by 105
On quantitative aspects of model interpretabilityPreprint
Nguyen, A.-P., & Martínez, M. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07584 · Cited by 179
“[3–7].”
A Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139
“Meanwhile, a refined taxonomy, distilled from existing surveys on XAI techniques, is presented below and used to categorize XAD methods.”
MethodInfluentialAyano, Y. M., Schwenker, F., Dufera, B. D., & Debelee, T. G. (2022) · Diagnostics · doi:10.3390/diagnostics13010111 · Cited by 167
“…the mathematical foundations in ML interpretable methods are not briefly discussed in these review works [36–38 Du et al. [39] and Carvalho et al. [40] presented the need that necessitates explaining the prediction of complex ML models by providing human-friendly explanations within societal…”
BackgroundMethodInfluentialCao, J. L., Zhou, T., Zhi, S., Lam, S., Ren, G., Zhang, Y., Wang, Y., Dong, Y., & Cai, J. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120212 · Cited by 141
Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions
Ibrahim, R., & Shafiq, M. O. (2022) · ACM Computing Surveys · doi:10.1145/3563691 · Cited by 161
NBDT: Neural-Backed Decision TreesPreprint
Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Jin, H., Petryk, S., Bargal, S. A., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00221 · Cited by 170
Explainable AI decision model for ECG data of cardiac disorders
Anand, A., Kadian, T., Shetty, M. K., & Gupta, A. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103584 · Cited by 154
Explainable Artificial Intelligence (XAI) in Insurance
Owens, E., Sheehan, B., Mullins, M., Cunneen, M., Ressel, J., & Castignani, G. (2022) · Risks · doi:10.3390/risks10120230 · Cited by 182
“The inclusion of the XAI variables and criteria is supported by previous research in XAI, with the criteria synthesised from Mueller et al. (2019); Du et al. (2019); Carvalho et al. (2019) and Payrovnaziri et al. (2020).”
BackgroundMethodInfluentialXAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131
Neto, M. P., & Paulovich, F. V. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030354 · Cited by 119
Nielsen, S. (2022) · Journal of Accounting & Organizational Change · doi:10.1108/jaoc-08-2020-0107 · Cited by 156
Analyzing Employee Attrition Using Explainable AI for Strategic HR Decision-Making
Díaz, G. M., Galán-Hernández, J. J., & Salvador, J. L. G. (2023) · Mathematics · doi:10.3390/math11224677 · Cited by 166
Khan, F. S., Mazhar, S. S., Mazhar, K., AlSaleh, D., & Mazhar, A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11215-9 · Cited by 150
Vale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124
“The internals of black-box models are either unknown or not clearly understood by humans [13, 32].”
BackgroundSHAP Value-Based Feature Importance Analysis for Short-Term Load Forecasting
Lee, Y.-G., Oh, J.-Y., Kim, D.-S., & Kim, G. (2022) · Journal of Electrical Engineering and Technology · doi:10.1007/s42835-022-01161-9 · Cited by 139
Lu, S.-C., Swisher, C. L., Chung, C. D., Jaffray, D. A., & Sidey-Gibbons, C. J. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1129380 · Cited by 150
“In such cases, an explanation in model decision-making process pertaining to erroneous or discriminative predictions enables model auditing, debugging, and re fi nement to ensure model performance and fairness (11, 21).”
BackgroundMethodInfluentialHassan, S. U., Abdulkadir, S. J., Zahid, M. S. M., & Al-Selwi, S. M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109569 · Cited by 108
SurvLIME: A method for explaining machine learning survival models
Kovalev, M. S., Utkin, L. V., & Kasimov, E. M. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106164 · Cited by 145
Operationalizing Human-Centered Perspectives in Explainable AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Mara, M., Streit, M., Wachter, S., Riener, A., & Riedl, M. O. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3441342 · Cited by 97
“The AI community has approached evaluation by defning normative, quantifable metrics, such as fdelity, completeness, stability, compactness, etc. [7, 26].”
BackgroundA review of explainable artificial intelligence in smart manufacturing
Madathil, A. P., Luo, X., Liu, Q., Walker, C. T., Madarkar, R., & Qin, Y. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2513574 · Cited by 146
“…it difficult for human operators to understand and CONTACT Xichun Luo xichun.luo@strath.ac.uk Centre for Precision Manufacturing, DMEM, University of Strathclyde, Glasgow G1 1XJ, UK trust their decision-making processes, especially in high-stakes environments (Carvalho, Pereira, and Cardoso 2019).”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“For interested readers, we suggest several recent papers that provided deeper technical surveys [6, 11, 20, 44].”
BackgroundDrozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
“nformation needed to establish initial trust, which may be different from information needed to maintain trust. 7 CONCLUSION As the AI industry is expected to grow significantly in the next few years [5], research on the relationship between user trust and automated data science systems is critical to ensure these too…”
BackgroundData-Driven Process Monitoring and Fault Diagnosis: A Comprehensive Survey
Melo, A., Câmara, M. M., & Pinto, J. C. (2024) · Processes · doi:10.3390/pr12020251 · Cited by 140
Machine learning for battery research
Wei, Z. X., He, Q., & Zhao, Y. (2022) · Journal of Power Sources · doi:10.1016/j.jpowsour.2022.232125 · Cited by 122
Machine Learning for Human Learners: Opportunities, Issues, Tensions and Threats
Webb, M. E., Fluck, A., Magenheim, J., Malyn-Smith, J., Waters, J., Deschênes, M., & Zagami, J. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-020-09858-2 · Cited by 116
“What is apparent however, is that there are no agreed metrics for the quality of explanation methods (Carvalho et al. 2019) and comparisons are difficult because explanations can vary in format so much across different contexts.”
BackgroundPractical early prediction of students’ performance using machine learning and eXplainable AI
Jang, Y., Choi, S., Jung, H., & Kim, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11120-6 · Cited by 114
“In other words, the interpretability of a machine learning model is high if a human can easily understand why it made a prediction (Carvalho et al., 2019).”
BackgroundA Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle
Shahriar, S., Allana, S., Hazratifard, S. M., & Dara, R. (2023) · IEEE Access · doi:10.1109/access.2023.3287195 · Cited by 114
“Utilizing inherently interpretable algorithms, such as decision trees, logistic regression, and linear regression, that include meaningful parameters to explain their predictions [157] can improve model transparency.”
MethodInterpretable machine learning for real estate market analysis
Lorenz, F., Willwersch, J., Cajias, M., & Fuerst, F. (2022) · Real Estate Economics · doi:10.1111/1540-6229.12397 · Cited by 129
Review of machine learning methods for RNA secondary structure prediction
Zhao, Q., Zhao, Z., Fan, X., Yuan, Z., Mao, Q., & Yao, Y. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009291 · Cited by 109
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 85
“Broadly, an AI system can be considered to be explainable (or XAI), if it aims to make its decisions easy to understand and interpret by people [12, 27, 48, 62, 66, 95, 108, 123].”
BackgroundA Practical Tutorial on Explainable AI Techniques
Bennetot, A., Donadello, I., El-Qadi, A., Dragoni, M., Frossard, T., Wagner, B., Saranti, A., Tulli, S., Trocan, M., Chatila, R., et al. (2024) · ACM Computing Surveys · doi:10.1145/3670685 · Cited by 131
“Model-agnostic XAI techniques are meant to be applied to any machine learning model and are applied after the model has been trained [18].”
A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, A., Besa, C., Uribe, S., Andía, M., Tejos, C., Prieto, C., & Capurro, D. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 95
“Providing interpretable justifications for the model’s outcome is essential in this medical domain, and furthermore, we should be able to evaluate them to answer questions such as, does the method justify the model’s decision?, which method provides a better explanation? However, there is no consensus on evaluation me…”
MethodArtificial intelligence-driven real-world battery diagnostics
Zhao, J., Qu, X., Wu, Y., Fowler, M., & Burke, A. F. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100419 · Cited by 100
Machine Learning: Models, Challenges, and Research Directions
Khoei, T. T., & Kaabouch, N. (2023) · Future Internet · doi:10.3390/fi15100332 · Cited by 121
Toward Trustworthy Artificial Intelligence (TAI) in the Context of Explainability and Robustness
chander, B., John, C., Warrier, L., & Gopalakrishnan, K. (2025) · ACM Computing Surveys · doi:10.1145/3675392 · Cited by 108
“The lack of robust AI systems harms model efficiency and faith in AI [81].”
Review on Interpretable Machine Learning in Smart Grid
Xu, C., Liao, Z., Li, C., Zhou, X., & Xie, R. (2022) · Energies · doi:10.3390/en15124427 · Cited by 126
“In-model interpretability aims to create ML models that are intrinsically interpretable [59].”
BackgroundMethodInfluentialHooshyar, D., & Yang, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3463948 · Cited by 135
“Finally, local interpretability for a group of predictions combines methods for individual and global interpretations, to explain predictions for multiple instances [42].”
MethodCiravegna, G., Barbiero, P., Giannini, F., Gori, M., Lió, P., Maggini, M., & Melacci, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103822 · Cited by 146
“However, the notion itself of explanation generally depends on both the application domain and whom it is aimed at [Carvalho et al., 2019].”
BackgroundResultMehdary, A., Chehri, A., Jakimi, A., & Saadane, R. (2024) · Sensors · doi:10.3390/s24041230 · Cited by 107
A Critical Survey on Fairness Benefits of Explainable AI
Deck, L., Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658990 · Cited by 142
“Thus, while the goal of these approaches is to report on procedural fairness, a true procedural perspective on fairness may only be provided by model-specific explanations [41], such as intrinsically interpretable rule lists (e.g., [8]).”
BackgroundExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
“Common goals of AI explanations, as reflected in much of the XAI literature, are to support a complete and sound understanding of the model [16, 48], and to foster trust in the AI [19, 69].”
BackgroundDeep learning technology for construction machinery and robotics
You, K., Zhou, C., & Ding, L. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104852 · Cited by 112
Cloud-Based Artificial Intelligence Framework for Battery Management System
Shi, D., Zhao, J., Eze, C., Wang, Z., Wang, J., Lian, Y., & Burke, A. F. (2023) · Energies · doi:10.3390/en16114403 · Cited by 96
Reviewing methods of deep learning for intelligent healthcare systems in genomics and biomedicine
Zafar, I., Anwar, S., kanwal, F., Yousaf, W., Nisa, F. U., Kausar, T., Ain, Q. U., Unar, A., Kamal, M. A., Rashid, S., et al. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105263 · Cited by 124
Wang, H., Shao, W., Sun, C., Yang, K., Cao, D., & Li, J. (2024) · Engineering · doi:10.1016/j.eng.2023.10.011 · Cited by 123
XAI for intrusion detection system: comparing explanations based on global and local scope
Hariharan, S., Robinson, R. R. R., Prasad, R. R., Thomas, C., & Balakrishnan, N. (2022) · Journal of Computer Virology and Hacking Techniques · doi:10.1007/s11416-022-00441-2 · Cited by 116
“To obtain the global explanation we have used Permutation Importance and SHAP explanation algorithms, and for local explanations, we have used LIME, SHAP and CIU [28].”
MethodOptimization of 4D/3D printing via machine learning: A systematic review
Alli, Y. A., Anuar, H., Manshor, M. R., Okafor, C. E., Kamarulzaman, A. F., Akçakale, N., Nazeri, F. N. M., Bodaghi, M., Suhr, J., & Nasir, N. A. M. (2024) · Hybrid Advances · doi:10.1016/j.hybadv.2024.100242 · Cited by 113
Eke, C. I., & Shuib, L. M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10868-x · Cited by 124
“To enhance the safety of AI systems, trans-parency makes it possible for them to be tested, audited, and debugged [63].”
(2020) · neural information processing systems · Cited by 100
Causal Discovery in Manufacturing: A Structured Literature Review
Vuković, M., & Thalmann, S. (2022) · Journal of Manufacturing and Materials Processing · doi:10.3390/jmmp6010010 · Cited by 143
“Understanding the reasoning behind the predictions made from ML models is crucial for their acceptance, especially in industrial scenarios [75].”
Fokkema, M., Edbrooke-Childs, J. H., & Wolpert, M. (2020) · Psychotherapy Research · doi:10.1080/10503307.2020.1785037 · Cited by 111
“As noted by Carvalho et al. (2019), there is no consensus on how to measure the quality of these explanations.”
BackgroundDeng, S., Su, J., Zhu, Y., Yu, Y., & Xiao, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122502 · Cited by 99
Lamane, H., Mouhir, L., Moussadek, R., Baghdad, B., Kişi, Ö., & Bilali, A. E. (2024) · International Journal of Sediment Research · doi:10.1016/j.ijsrc.2024.10.002 · Cited by 112
XGBoost-Based Framework for Smoking-Induced Noncommunicable Disease Prediction
Davagdorj, K., Pham, V.-H., Theera‐Umpon, N., & Ryu, K. H. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186513 · Cited by 100
“Interpretability methods can be categorized into three types: explain data, build an inherently interpretable model (in modeling), and allow to explain it after building the models [43].”
MethodThe Role of Causality in Explainable Artificial Intelligence
Carloni, G., Berti, A., & Colantonio, S. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70015 · Cited by 127
“…in Van Lent et al. (2004), but its popularity has spread across the literature only after the DARPA’s XAI program (Gunning and Aha, 2019), reaching a certain degree of maturity to date (Guidotti et al., 2018; Du et al., 2019; Carvalho et al., 2019; Rudin, 2019; Arrieta et al., 2020; Molnar, 2020).”
State of the Art of Visual Analytics for eXplainable Deep Learning
Rosa, B. L., Blasilli, G., Bourqui, R., Auber, D., Santucci, G., Capobianco, R., Bertini, E., Giot, R., & Angelini, M. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14733 · Cited by 117
“In literature, there is no consensus about the difference between the terms explainability and interpretability, and their definitions [CPC19, AB18].”
BackgroundRiveiro, M., & Thill, S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103507 · Cited by 79
A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning
Sperrle, F., El‐Assady, M., Guo, G., Borgo, R., Chau, D. H., Endert, A., & Keim, D. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14329 · Cited by 75
Diwali, A., Saeedi, K., Dashtipour, K. K., Gogate, M., Cambria, E., & Hussain, A. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3296373 · Cited by 111
“The basic concept is to focus on a specific example to clarify how the model’s prediction was arrived at [53].”
BackgroundViCE: visual counterfactual explanations for machine learning models
Gomez, O., Holter, S., Yuan, J., & Bertini, E. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377536 · Cited by 91
“Previous works [1–4, 6, 10, 15, 17] provide an overview of methods, opportunities and challenges in this area.”
Toward interpretable LSTM-based modeling of hydrological systems
Fuente, L. D. L., Ehsani, M. R., Gupta, H. V., & Condon, L. E. (2024) · Hydrology and earth system sciences · doi:10.5194/hess-28-945-2024 · Cited by 105
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Wang, H., Yan, R., Wang, S., & Zhen, L. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104361 · Cited by 90
Navigating through the Maze of Homogeneous Catalyst Design with Machine Learning
Gomes, G., Pollice, R., & Aspuru‐Guzik, A. (2021) · Trends in Chemistry · doi:10.1016/j.trechm.2020.12.006 · Cited by 94
Olier, I., Ortega‐Martorell, S., Pieroni, M., & Lip, G. Y. H. (2021) · Cardiovascular Research · doi:10.1093/cvr/cvab169 · Cited by 94
Toward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 111
“Others, however, adopt different distinctions between the terms [38], do not differentiate between both concepts [34], [37], [39], or use other terms such as “intelligibility” and “understandability” interchange-ably [47].”
BackgroundMethodTamagusko, T., Correia, M. G., & Ferreira, A. (2024) · Infrastructures · doi:10.3390/infrastructures9120213 · Cited by 99
“An increasing amount of research concentrates on interpretability and explainability in ML models [178–181].”
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
“This will require progress on both developing more intrinsically understandable systems and techniques to interpret trained ones [45, 256].”
BackgroundThe Final Frontier: Deep Learning in Space
Kothari, V., Liberis, E., & Lane, N. D. (2020) · Proceedings of the 21st International Workshop on Mobile Computing Systems and Applications · doi:10.1145/3376897.3377864 · Cited by 86
“such as LIME or contextuality models [38] allows for humans-inthe-loop systems both in V&V and mission control.”
MethodMoreno-Sánchez, P. A. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1219586 · Cited by 101
“In the medical context, the lack of explainability of certain prediction models must be addressed since clinicians find challenging to trust complex ML methods due to their high technical knowledge requirements (Carvalho et al., 2019).”
BackgroundNeuro-Fuzzy Random Vector Functional Link Neural Network for Classification and Regression Problems
Sajid, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3359652 · Cited by 95
“Considering decision trees as one of the best interpretable models [45], we compare the NF-RVFL and standard RVFL models with the decision tree.”
MethodMATHEMATICS FOR FINANCE: A REVIEW OF QUANTITATIVE METHODS IN LOAN PORTFOLIO OPTIMIZATION
Kowsar, M. M., Mohiuddin, M., & Islam, S. (2023) · doi:10.63125/j43ayz68 · Cited by 86
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Hicks, S. A., Isaksen, J. L., Thambawita, V., Ghouse, J., Ahlberg, G., Linneberg, A., Grarup, N., Strümke, I., Ellervik, C., Olesen, M. S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-90285-5 · Cited by 77
Explainable Deep Learning for Personalized Age Prediction With Brain Morphology
Lombardi, A., Diacono, D., Amoroso, N., Monaco, A., Tavares, J. M. R. S., Bellotti, R., & Tangaro, S. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.674055 · Cited by 103
“Understanding how the models affect the decisions and how each feature is related to the outcomes can increase confidence in the models and broaden their applications in the clinical setting (Carvalho et al., 2019; Holzinger et al., 2019).”
BackgroundZero-touch networks: Towards next-generation network automation
Rajab, M. E., Yang, L., & Shami, A. A. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110294 · Cited by 96
Reimann, R., Zeng, B., Jakopec, M., Burdukiewicz, M., Petrick, I., Schierack, P., & Rödiger, S. (2020) · Algal Research · doi:10.1016/j.algal.2020.101908 · Cited by 79
Interpretable Machine Learning for Discovery: Statistical Challenges and Opportunities
Allen, G. I., Gan, L., & Zheng, L. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-030919 · Cited by 115
Hamida, S. U., Chowdhury, M. J. M., Chakraborty, N. R., Biswas, K., & Sami, S. K. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110149 · Cited by 95
Explainability of Machine Learning Models for Bankruptcy Prediction
Park, M. S., Son, H., Hyun, C., & Hwang, H. J. (2021) · IEEE Access · doi:10.1109/access.2021.3110270 · Cited by 83
“Although there are many studies regarding the interpretability of machine learning algorithms (e.g., [18]–[20]), we focus on an instance-wise local interpretation method.”
BackgroundAn interpretable forecasting framework for energy consumption and CO2 emissions
Aras, S., & Van, M. H. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120163 · Cited by 89
Time series big data: a survey on data stream frameworks, analysis and algorithms
Almeida, A., Brás, S., Sargento, S., & Pinto, F. C. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00760-1 · Cited by 100
Reinforcement Learning Interpretation Methods: A Survey
Alharin, A., Doan, T.-N., & Sartipi, M. (2020) · IEEE Access · doi:10.1109/access.2020.3023394 · Cited by 94
“RL interpretation can be performed in three different ways [26]: using another method to generate explanations while keeping the original model (Post-Hoc), introducing a new interpretable learning model (Intrinsic), or a combination both methods by altering the original model to improve the…”
MethodInfluentialRaab, D., Theissler, A., & Spiliopoulou, M. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07809-x · Cited by 94
“Driven by the increased popularity of ML systems in various domains, products and services [5–7] and the resulting significant impact on society [8], explaining and interpreting these systems has become a crucial skill.”
BackgroundPelegrina, G. D., Duarte, L. T., & Grabisch, M. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.104014 · Cited by 97
Yun, K. K., Yoon, S. W., & Won, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118803 · Cited by 94
Gurmessa, D. K., & Jimma, W. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100954 · Cited by 65
“11 Therefore, using machine learning explanations can increase the transparency, interpretability, fairness, robustness, privacy, trust and reliability of machine learning models.”
MethodHuang, G., Li, Y., Jameel, S., Long, Y., & Papanastasiou, G. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.05.004 · Cited by 99
“By carefully investigating XIAI DL in healthcare NLP, our review aims to provide insights into the scientific and clinical impact that can potentially be important for XIAI democratization in healthcare NLP.”
BackgroundMethodThe Different Faces of AI Ethics Across the World: A Principle-to-Practice Gap Analysis
Tidjon, L. N., & Khomh, F. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3225132 · Cited by 88
“…[136], [137], [138], security [126], [136], [139], [140], robustness [127], [141], [142], [143], [144], [145], [146], explain-ability [147], [148], [149], interpretability [147], [148], [149], transparency and accountability [147], [148], [149], [150], traceability and auditability [129], [151],…”
Alomari, Y., & Andó, M. (2024) · Results in Engineering · doi:10.1016/j.rineng.2024.101834 · Cited by 84
Nascita, A., Aceto, G., Ciuonzo, D., Montieri, A., Persico, V., & Pescapè, A. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3504955 · Cited by 98
“Post-hoc explainability refers to methods that analyze and interpret a pre-trained AI model [40].”
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78
Barbado, A., Corcho, Ó., & Benjamins, R. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116100 · Cited by 77
“There are some recent reviews in the literature that deal with the challenge of providing metrics in XAI, such as [29].”
BackgroundExplaining CNN and RNN Using Selective Layer-Wise Relevance Propagation
Jung, Y.-J., Han, S.-H., & Choi, H.-J. (2021) · IEEE Access · doi:10.1109/access.2021.3051171 · Cited by 68
Combining Machine Learning with Physical Knowledge in Thermodynamic Modeling of Fluid Mixtures
Jirasek, F., & Hasse, H. (2023) · Annual Review of Chemical and Biomolecular Engineering · doi:10.1146/annurev-chembioeng-092220-025342 · Cited by 87
A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities
Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M., & Yang, J. (2024) · ACM Computing Surveys · doi:10.1145/3665926 · Cited by 98
“ML explainability is the field interested in developing post-hoc (explainability) methods and (inherently explainable) models that allow the internal functioning of ML systems to be understandable to humans [39].”
Viana, C. M., Santos, M., Freire, D., Abrantes, P. C. D. R. M., & Rocha, J. (2021) · Ecological Indicators · doi:10.1016/j.ecolind.2021.108200 · Cited by 79
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.