Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
“According to Carvalho et al. (2019), such transparency is pivotal for enhancing understanding of AI dialogue systems, laying a solid foundation for the challenges and advantages of algorithmic decisions, and ensuring that decision-making processes are both accountable and fair.”
BackgroundInfluentialExplainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
“, model accuracy) do not always exhibit true prediction decisions [9].”
BackgroundA comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127
“Interpretabil-ity (Carvalho et al., 2019) refers to the ability of a user to understand the ensemble outcomes.”
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983
Ponce‐Bobadilla, A. V., Schmitt, V., Maier, C. S., Mensing, S., & Stodtmann, S. (2024) · Clinical and Translational Science · doi:10.1111/cts.70056 · Cited by 826
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
“he development of intelligent systems that optimize the choice of explanation, such as interactive or personalized explanations [4,81,94]. Toward this goal, many sought to define the desiderata of XAI [19,40,45,77], including fi- delity, completeness, robustness, etc. Despite the conceptual discussions, there are few es…”
BackgroundMethodInfluentialZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
AI Alignment: A Contemporary Survey
Ji, J., Qiu, T., Chen, B., Zhou, J., Zhang, B., Hong, D., Lou, H., Wang, K., Duan, Y., He, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3770749 · Cited by 802
“The coherent extrapolated volition concept was developed to formalize the role of reflection in shaping human moral values [188].”
InfluentialCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K., Dickerson, J., & Shah, C. (2024) · ACM Computing Surveys · doi:10.1145/3677119 · Cited by 255
“Several works have summarized and reviewed other research in explainable machine learning [3, 51, 127].”
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
“Researchers have leveraged the feature importance to explain the characteristics and significance of the extracted features and the correlations among features and between features and the outcomes for providing interpretability for AI models [2,141,142].”
BackgroundMethodTrust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
“Besides, a consensual de fi nition for interpretability in AI and how to quantify it has not yet been reached (Carvalho et al., 2019; Doshi-Velez and Kim, 2017; Molnar, 2020).”
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
“Research-wise, the field is maturing in terms of methods surveys [75,41,120,96,1,6,23,15], further consolidation of terms and knowledge [42,22,82,97,88,17], and work about defining interpretability or evaluation of IML methods [74,73,95,49].”
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
“The “right” level of explainability required depends on many factors and is context and resource specific [138].”
BackgroundLongo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“In other words, despite the broad interest in the design of XAI methods [18,31,48–50], it is still unclear how to compare the results of different evaluations and establish a common understanding of how to evaluate explanations.”
BackgroundEvaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606
“However, most of the work in the ML explanation research field focuses on creating new methods and techniques to improve explainability in prediction tasks and try to minimise the decrease in the prediction accuracy [17].”
BackgroundInfluentialUcar, A., Karakose, M., & Kırımça, N. (2024) · Applied Sciences · doi:10.3390/app14020898 · Cited by 444
“[162] also classified the interpretation techniques as local and global.”
BackgroundNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“1Regarding terminology: ‘interpretability’ and ‘explainability’ are closely related and often used interchangeably in the XAI context [29, 32].”
BackgroundMethodInfluentialCounterfactual explanations and how to find them: literature review and benchmarking
Guidotti, R. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00831-6 · Cited by 471
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Researchers [29, 47, 81, 101, 107, 183] also proposed many other desiderata for interpretations, interpretation algorithms or interpretability, such as fairness, privacy, reliability, robustness, causality, trust, fidelity, faithfulness, transferability, informativeness, transparency, plausibility, satisfaction, accou…”
BackgroundMethodInfluentialOpening the black box: the promise and limitations of explainable machine learning in cardiology
Petch, J., Di, S., & Nelson, W. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.09.004 · Cited by 552
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
“With the rise of opaque and poorly-understood machine learning models in the field of AI, trust is often cited as a key desirable property of the interaction between any user and the AI [8, 11, 71, 76].”
BackgroundInterpretability of machine learning-based prediction models in healthcare
Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., & Cilar, L. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1379 · Cited by 541
“There are different criteria for classifying methods for machine learning interpretability such as intrinsic or post-hoc classification, pre-model, in-model or post-model and classification based on the model outcome (Molnar, 2018; Carvalho, Pereira & Cardoso, 2019).”
BackgroundMethodExplainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
“Although there is no established consensus on the complete set of factors that makes an AI system explainable, XAI work commonly shares the goal of making an AI system’s functioning or decisions easy to understand by people [7, 14, 27, 31, 34, 62, 70, 82].”
BackgroundSalahuddin, Z., Woodruff, H. C., Chatterjee, A., & Lambin, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105111 · Cited by 508
Ferreira, C., & Gonçalves, G. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.05.010 · Cited by 417
A Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing · doi:10.18653/v1/2020.aacl-main.46 · Cited by 203
“We refer interested readers to (Carvalho et al., 2019) for a comprehensive presentation of aspects of evaluating approaches.”
BackgroundExplainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
Zhang, Z., Hamadi, H. A., Damiani, E., Yeun, C. Y., & Taher, F. (2022) · IEEE Access · doi:10.1109/access.2022.3204051 · Cited by 450
“CATEGORIZATIONS OF XAI According to [53], [54], the XAI categories can be structured in a variety of aspects shown in Figure 6.”
BackgroundHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 472
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
Speith, T. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534639 · Cited by 337
Schwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328
“Faithfulness [84] or fidelity [23], soundness [143], causality [22], measures how accurately the behavior of the explantor conforms with that of the actual object of explanation.”
BackgroundInfluentialBenchmarking and survey of explanation methods for black box models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00933-9 · Cited by 264
“Further taxonomies and definitions are presented in (Murdoch et al. 2019; Carvalho et al. 2019).”
BackgroundInfluentialThe Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Krishna, S., Han, T., Gu, A., Jabbari, S., Wu, Z. S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2963888/v1 · Cited by 201
Mahbooba, B., Timilsina, M., Sahal, R., & Serrano, M. (2021) · Complexity · doi:10.1155/2021/6634811 · Cited by 327
“Explanation methods and techniques for AI/ML interpretability can be classified according to different criteria as follows: (1) premodel, in-model, and postmodel, (2) intrinsic and post hoc, (3) model-specific and model-agnostic, (4) feature engineering, and so on [8].”
BackgroundMethodInherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Payrovnaziri, S. N., Chen, Z., Rengifo-Moreno, P., Miller, T., Bian, J., Chen, J. H., Liu, X., & He, Z. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa053 · Cited by 313
“74,75 Figure 3 depicts the type of XAI methods employed along with different ML methods and Table 3 summarizes different approaches in each category of XAI methods for each of the articles.”
MethodExplainable Artificial Intelligence in CyberSecurity: A Survey
Capuano, N., Fenza, G., Loia, V., & Stanzione, C. (2022) · IEEE Access · doi:10.1109/access.2022.3204171 · Cited by 330
“Carvalho et al. [33] add a criterion on the stage of model 402 development, in-model interpretability that concerns ML 403 models that have inherent interpretability in it (through con-404 straints or not).”
BackgroundExplainable Reinforcement Learning: A Survey
Puiutta, E., & Veith, E. M. S. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_5 · Cited by 252
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302
Interpretable machine learning with an ensemble of gradient boosting machines
Konstantinov, A. V., & Utkin, L. V. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106993 · Cited by 271
Machine learning in materials science: From explainable predictions to autonomous design
Pilania, G. (2021) · Computational Materials Science · doi:10.1016/j.commatsci.2021.110360 · Cited by 281
Combustion machine learning: Principles, progress and prospects
Ihme, M., Chung, W. T., & Mishra, A. A. (2022) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2022.101010 · Cited by 265
Ghannam, R. B., & Techtmann, S. M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.01.028 · Cited by 278
“Often the learning algorithms applied to marker-gene datasets are developed and implemented for improved performance, rather than for model interpretation [104,105].”
BackgroundMethodRelation between prognostics predictor evaluation metrics and local interpretability SHAP values
Baptista, M. L., Goebel, K., & Henriques, E. M. P. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103667 · Cited by 270
“[23], consider the existence of pre-model, or data, interpretability.”
BackgroundMethodInfluentialKiseleva, A., Kotzinos, D., & De Hert, P. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.879603 · Cited by 278
“…in legislation and in legal literature: accountability (Felzmann et al., 2020; Rieder and Hofmann, 2020); ensuring efficacy, safety, and quality (Carvalho et al., 2019); improving trust (Kiseleva, 2020a In this paper, we propose these functions of transparency: 1) accountability (that, inter…”
BackgroundResultInfluentialTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“From these studies, we distill three main research questions concerning the effects of model explanations on each objective.”
BackgroundPost-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207
Interpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Hands-On Machine Learning with R · doi:10.1201/9780367816377-16 · Cited by 230
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268
“Besides these three criteria, Carvalho et al. [15] add a criterion on the stage of model development where an explainability method is applied, i.e., whether it is before ( pre-model ), during ( in-model ), or after ( post-model ) building the machine learning model.”
BackgroundMethodCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267
“Several works have summarized and reviewed other research in explainable machine learning [11, 22, 58].”
BackgroundMoustafa, N., Koroniotis, N., Keshk, M., Zomaya, A. Y., & Tari, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3280465 · Cited by 266
Application of machine learning algorithms in municipal solid waste management: A mini review
Xia, W., Jiang, Y., Chen, X., & Zhao, R. (2022) · Waste Management & Research: The Journal for a Sustainable Circular Economy · doi:10.1177/0734242x211033716 · Cited by 240
“However, in the past few years, there has been a lot of work in computer science to make ML models interpretable (Carvalho et al., 2019), but in the field of MSWM, such research is still rare.”
BackgroundSocially Responsible AI Algorithms: Issues, Purposes, and Challenges
Cheng, L., Varshney, K. R., & Liu, H. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12814 · Cited by 261
A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 203
“Using feature importance as a tool, researchers have been able to better describe the properties and importance of extracted features, as well as the relationships between those features and the outcomes they predict [12], [113], [127].”
MethodRecent Applications of Explainable AI (XAI): A Systematic Literature Review
Saarela, M., & Podgorelec, V. (2024) · Applied Sciences · doi:10.3390/app14198884 · Cited by 183
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Theissler, A., Spinnato, F., Schlegel, U., & Guidotti, R. (2022) · IEEE Access · doi:10.1109/access.2022.3207765 · Cited by 194
“…reviewed work 3) insights into the differences and advantages of such explainable AI techniques; 4) highlights of applications and evaluation strategies to showcase applied XAI techniques; 5) research directions in order to inspire future research in the field of XAI for time series classification.”
BackgroundInfluentialEnnab, M., & Mcheick, H. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1444763 · Cited by 238
Perturbation-based methods for explaining deep neural networks: A survey
Ivanovs, M., Kadikis, R., & Ozols, K. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.030 · Cited by 204
Machine Learning Toxicity Prediction: Latest Advances by Toxicity End Point
Cavasotto, C. N., & Scardino, V. (2022) · ACS Omega · doi:10.1021/acsomega.2c05693 · Cited by 213
Maintenance optimization in industry 4.0
Pinciroli, L., Baraldi, P., & Zio, E. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109204 · Cited by 185
Counterfactual Explanations for Machine Learning: A Review.Preprint
Verma, S., Dickerson, J. P., & Hines, K. (2020) · arXiv (Cornell University) · Cited by 332
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
“uncertainty awareness [18, 73], we avoid using the term “uncertainty” as participants’ comments reflect their subjective perception of reliability rather than the actual model uncertainty quantity.”
MethodDing, W., Abdel-Basset, M., Hawash, H., & Ali, A. M. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.013 · Cited by 183
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Interpretable and explainable machine learning: A methods‐centric overview with concrete examples
Marcinkevičs, R., & Vogt, J. E. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1493 · Cited by 161
“Nevertheless, the notion of interpretability often depends on the domain of application (Rudin, 2019) and the target explainee (Carvalho et al., 2019), that is, the recipient of interpretations and explanations.”
BackgroundMethodInfluentialCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
ElShawi, R., Sherif, Y., Al‐Mallah, M., & Sakr, S. (2021) · Computational Intelligence · doi:10.1111/coin.12410 · Cited by 177
Hao, X., Demir, E., & Eyers, D. R. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102662 · Cited by 229
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“EXPLAINATION EVALUATION Evaluation of explanations with metrics and methods is a major point of concern to leverage explainability and user confidence as they are task-specific and subjective [25].”
BackgroundRyo, M. (2022) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2022.11.003 · Cited by 166
Ryo, M., Angelov, B., Mammola, S., Kass, J. M., Benito, B. M., & Hartig, F. (2021) · Ecography · doi:10.1111/ecog.05360 · Cited by 182
“This problem has led to the emerging research area of explainable artificial intelligence (xAI), a subfield of AI also termed interpretable ML (Murdoch et al. 2019), that aims at developing tools for enhancing the interpretability of complex algorithms (Carvalho et al. 2019).”
BackgroundFruit ripeness classification: A survey
Rizzo, M., Marcuzzo, M., Zangari, A., Gasparetto, A., & Albarelli, A. (2023) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2023.02.004 · Cited by 167
Roadmap on Machine learning in electronic structure
Kulik, H. J., Hammerschmidt, T., Schmidt, J., Botti, S., Marques, M. A. L., Boley, M., Scheffler, M., Todorović, M., Rinke, P., Oses, C., et al. (2022) · Electronic Structure · doi:10.1088/2516-1075/ac572f · Cited by 180
Explainability of a Machine Learning Granting Scoring Model in Peer-to-Peer Lending
Ariza-Garzon, M. J., Arroyo, J., Caparrini, A., & Segovia-Vargas, M.-J. (2020) · IEEE Access · doi:10.1109/access.2020.2984412 · Cited by 172
“Carvalho et al. [55] review inter-pretability from an ontological and epistemological perspective.”
BackgroundExplainable Artificial Intelligence (XAI) for Internet of Things: A Survey
Kök, İ., Okay, F. Y., Muyanlı, Ö., & Özdemir, S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3287678 · Cited by 190
“The taxonomy of XAI can be classified from a variety of perspectives [12], [30], as shown in Fig.”
ChatGPT is a Remarkable Tool—For Experts
Azaria, A., Azoulay, R., & Reches, S. (2023) · Data Intelligence · doi:10.1162/dint_a_00235 · Cited by 237
Argumentation and explainable artificial intelligence: a survey
Vassiliades, A., Bassiliades, N., & Patkos, T. (2021) · The Knowledge Engineering Review · doi:10.1017/s0269888921000011 · Cited by 160
“A survey that gathers methods to explain the decision of a deep learning model, as well as the open challenges, can be found in Carvalho et al .”
Wei, L., Niraula, D., Gates, E. D. H., Fu, J., Luo, Y., Nyflot, M. J., Bowen, S. R., El Naqa, I. M., & Cui, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230211 · Cited by 162
Interpretability and Explainability: A Machine Learning Zoo Mini-tourPreprint
Marcinkevičs, R., & Vogt, J. E. (2020) · arXiv (Cornell University) · doi:10.3929/ethz-b-000454597 · Cited by 221
Machine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174
“[174,175] These explanation methods are independent of the model itself. they rely on the analysis of features and outputs to perform the interpretation of the model, can be used for the explanation of black box models.”
MethodPham, L. T., Luo, L., & Finley, A. O. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-2997-2021 · Cited by 182
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024) · JMIR Medical Informatics · doi:10.2196/50048 · Cited by 172
“Recent advancements in ML interpretability have also been subject to review [18].”
BackgroundInterpretability research of deep learning: A literature survey
Xu, B., & Yang, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102721 · Cited by 211
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170
Review Study of Interpretation Methods for Future Interpretable Machine Learning
Mi, J.-X., Li, A.-D., & Zhou, L.-F. (2020) · IEEE Access · doi:10.1109/access.2020.3032756 · Cited by 158
“For the classification of interpretable methods, most of the articles are based on the intrinsic inter-pretable model and Post-hoc explanation [7], [8].”
BackgroundMethodUsing Machine Learning to Predict Complications in Pregnancy: A Systematic Review
Bertini, A., Salas, R., Chabert, S., Sobrevía, L., & Pardo, F. (2022) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2021.780389 · Cited by 197
“Interpretability is important before any health-care team can increase reliance on ML systems (Carvalho et al., 2019).”
BackgroundThe coming of age of interpretable and explainable machine learning models
Lisboa, P. J. G., Saralajew, S., Vellido, A., Fernández-Domenech, R., & Villmann, T. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.02.040 · Cited by 198
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 223
“Several recent reviews have explored specific topics related to LLMs, such as fairness and bias [11], interpretability [12], explainability [13], and privacy preservation [14].”
BackgroundMulti-Objective Hyperparameter Optimization in Machine Learning—An Overview
Karl, F., Pielok, T., Moosbauer, J., Pfisterer, F., Coors, S., Binder, M., Schneider, L., Thomas, J., Richter, J., Lang, M., et al. (2023) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3610536 · Cited by 204
“For example, the former is required to satisfy regulatory constraints in the banking sector [Bücker et al. 2022 ; Carvalho et al. 2019 ] and can be thought of as a constraint on the search space when selecting a model.”
BackgroundTrustworthy AI and robotics: Implications for the AEC industry
Emaminejad, N., & Akhavian, R. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104298 · Cited by 174
“Technologies that involve obscure processes between the input fed to the system and the generated output are generally less trustworthy than those with transparent and interpretable inner-workings (Carvalho, Pereira et al. 2019).”
BackgroundAbdullah, T. A. A., Zahid, M. S. M., & Ali, W. (2021) · Symmetry · doi:10.3390/sym13122439 · Cited by 178
Implementing machine learning in medicine
Verma, A. A., Murray, J., Greiner, R., Cohen, J., Shojania, K. G., Ghassemi, M., Straus, S. E., Pou-Prom, C., & Mamdani, M. (2021) · Canadian Medical Association Journal · doi:10.1503/cmaj.202434 · Cited by 172
Hermosilla, P., Berríos, S., & Allende, H. (2025) · Applied Sciences · doi:10.3390/app15137329 · Cited by 153
“Jaccard similarity has gained traction in comparative studies of XAI methods, quantifying overlap in the top explanatory features across methods or instances [41].”
Adversarial XAI Methods in Cybersecurity
Kuppa, A., & Le‐Khac, N. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3117075 · Cited by 170
Application of machine learning approaches in supporting irrigation decision making: A review
Umutoni, L., & Samadi, V. S. (2024) · Agricultural Water Management · doi:10.1016/j.agwat.2024.108710 · Cited by 195
A Survey on Image-Based Automation of CCTV and SSET Sewer Inspections
Haurum, J. B., & Moeslund, T. B. (2020) · Automation in Construction · doi:10.1016/j.autcon.2019.103061 · Cited by 153
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.