Machine Learning Interpretability: A Survey on Methods and Metrics
Open accessThe DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index.
Abstract
Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transparency enables AI systems to be tested, audited, and debugged, which is a path toward increasing their safety (Carvalho et al., 2019)15.”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“In this paper, we follow scholars who view “interpretability” as the broader concept that, inter alia, includes “explainability.” Specifically, we agree with the statement of M. Honneger that “interpretability is the end-goal we want to achieve, and explanations are the tools to reach interpretability” (Honegger, 2018, citation in Carvalho et al., 2019).”
From Transparency of AI in Healthcare as a Multilayered System of Accountabilities: Between Legal Requirements and Technical Limitations (2022) · cites this paper as Carvalho et al., 2019
In the citing paper
“These methods can capture complex, non-linear interactions among multiple factors that traditional models may miss [29,30].”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [29]
In the citing paper
“The required auditability property of Trustworthy AI systems demands the development of practical tools [109] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [110], as well as metrics beyond explainability [111-115], such as on traceability, data quality and integrity.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [111]
In the citing paper
“Model performance metrics (e.g., model accuracy) do not always exhibit true prediction decisions [9].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [9]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“For example, introduced in 2016, deep residual networks (ResNets) [1] are over 200-layers deep and have been shown to beat human-level performance on object recognition tasks.”
Background(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Partial Dependence Plot [142] Global Feature summary Individual Condition Expectation [143] Global/Local Feature summary Accumulated Local Effects Plot [144] Global Feature summary Feature Interaction [145] Global Feature summary Feature Importance [146] Global/Local Feature summary Local Surrogate Model [98] Local Su…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 3×
“Shapley Values is sometimes called by SHAP.”
Background"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · cited 1×
“These techniques range from classic descriptive statistics to data visualization methods, including Principal Component Analysis (PCA) [101] and t-SNE [102] (t-Distributed Stochastic Neighbor Embedding), and clustering methods, such as k-means [103] and MMD-critic [96] (Maximum Mean Discrepancy).”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Research in this type of methods exists for some years [134] but recently increased along with the ML interpretability field [135–137].”
MethodDavid, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Research works printed but not linked: 24
- Tractica (2019, January 22). Artificial Intelligence Software Market to Reach $105.8 Billion in Annual Worldwide Revenue by 2025. Available online: https://www.tractica.com/newsroom/press-releases/artificial-intelligence-software-market-to-reach-105-8-billion-in-annual-worldwide-revenue-by-2025/.
- Du, M., Liu, N., and Hu, X. (2018). Techniques for Interpretable Machine Learning. arXiv.
- Committee on Technology National Science and Technology Council and Penny Hill Press (2016). Preparing for the Future of Artificial Intelligence, CreateSpace Independent Publishing Platform.
- ACM US Public Council (2019, January 22). Statement on Algorithmic Transparency and Accountability. Available online: https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
- Kim, B., Malioutov, D.M., and Varshney, K.R. (2016). Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). arXiv.
- Kim, B., Malioutov, D.M., Varshney, K.R., and Weller, A. (2017). Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017). arXiv.
- Kim, B., Varshney, K.R., and Weller, A. (2018). Proceedings of the 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018). arXiv.
- Wilson, A.G., Kim, B., and Herlands, W. (2016). Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems. arXiv.
- Caruana, R., Herlands, W., Simard, P., Wilson, A.G., and Yosinski, J. (2017). Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning. arXiv.
- Pereira-Fariña, M., and Reed, C. (2017). Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Association for Computational Linguistics (ACL).
- IJCAI (2019, July 12). IJCAI 2018—Workshop on Explainable Artificial Intelligence (XAI). Available online: http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/.
- CD-MAKE (2019, July 12). CD-MAKE 2019 Workshop on explainable Artificial Intelligence. Available online: https://cd-make.net/special-sessions/make-explainable-ai/.
- Lim, B., Smith, A., and Stumpf, S. (2018). ExSS 2018: Workshop on Explainable Smart Systems. CEUR Workshop Proceedings, City, University of London Institutional Repository. Available online: http://openaccess.city.ac.uk/20037/.
- ICAPS (2019, July 12). ICAPS 2018—Workshop on Explainable AI Planning (XAIP). Available online: http://icaps18.icaps-conference.org/xaip/.
- ICAPS (2019, July 12). ICAPS 2019—Workshop on Explainable AI Planning (XAIP). Available online: https://kcl-planning.github.io/XAIP-Workshops/ICAPS_2019.
- CVPR (2019, July 12). CVPR 19—Workshop on Explainable AI. Available online: https://explainai.net/.
- Lipton, Z.C. (2016). The mythos of model interpretability. arXiv.
- Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv.
- Kim, T.W. (2018). Explainable Artificial Intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv.
- Ross, A., Lage, I., and Doshi-Velez, F. (2017, January 4–9). The neural lasso: Local linear sparsity for interpretable explanations. Proceedings of the Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
- Lee, M., He, X., Yih, W.t., Gao, J., Deng, L., and Smolensky, P. (2015). Reasoning in vector space: An exploratory study of question answering. arXiv.
- Xu, K., Park, D.H., Yi, C., and Sutton, C. (2018). Interpreting Deep Classifier by Visual Distillation of Dark Knowledge. arXiv.
- Koh, P.W., and Liang, P. (2017). Understanding black-box predictions via influence functions. arXiv.
- Bonnans, J.F., and Shapiro, A. (2013). Perturbation Analysis of Optimization Problems, Springer Science & Business Media.
Cited by
Items 601–700 of 1,007
Chen, Q., Pan, G., Chen, W., & Wu, P. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3084846 · Cited by 23
Correct-by-Construction Runtime Enforcement in AI – A Survey
Könighofer, B., Bloem, R., Ehlers, R., & Pek, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22337-2_31 · Cited by 28
Applying AI in Practice: Key Challenges and Lessons Learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobieczky, F., Zellinger, W., & Moser, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_25 · Cited by 21
Interpreting Deep Neural Networks Through Variable ImportancePreprint
Ish-Horowicz, J., Udwin, D., Flaxman, S. R., Filippi, S., & Crawford, L. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09839 · Cited by 24
A Framework to Learn with InterpretationPreprint
Parekh, J., Mozharovskyi, P., & d’Alché–Buc, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09345 · Cited by 21
Historical Perspectives in Volatility Forecasting Methods with Machine Learning
Qiu, Z., Kownatzki, C., Scalzo, F., & Cha, E. S. (2025) · Risks · doi:10.3390/risks13050098 · Cited by 22
“There are several ways to address this problem, as pointed out by Carvalho et al. (2019).”
Explainable person–job recommendations: challenges, approaches, and comparative analysis
Tang, F., Zhu, R., Yao, F., Wang, J., Luo, L., & Li, B. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1660548 · Cited by 24
“Studies have investigated black boxes and explainability issues in general machine learning (ML) and AI systems (Carvalho et al., 2019; Mi et al., 2020; Brasse et al., 2023; Marcinkevics and Vogt, 2023; Hassija et al., 2024).”
Rahman, M. (2026) · American Journal of Interdisciplinary Studies · doi:10.63125/np6bw920 · Cited by 17
Predicting the Appropriate Mode of Childbirth using Machine Learning Algorithm
Kowsher, M., Prottasha, N. J., Tahabilder, A., Habib, K., Abdur-Rakib, M., & Alam, M. S. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120582 · Cited by 17
Dauphin, J.-F., Dybczak, K., Maneely, M., Sanjani, M. T., Suphaphiphat, N., Wang, Y., & Zhang, H. (2022) · IMF Working Paper · doi:10.5089/9798400204425.001 · Cited by 37
“On the other hand, the research on the interpretability of ML algorithms has been scarce so far (Carvalho et al 2019) and ML algorithms are often seen as black boxes with a clear trade-off between model performance and interpretability (Linardatos et al 2020).”
BackgroundA Review of Neural Network-Based Emulation of Guitar Amplifiers
Vanhatalo, T., Legrand, P., Desainte‐Catherine, M., Hanna, P., Brusco, A., Pille, G., & Bayle, Y. (2022) · Applied Sciences · doi:10.3390/app12125894 · Cited by 17
“Although methods of increasing interpretability and explainability in DL are being studied, a range of methods are presented in [60] for example, and the advent of both gray-and white-box methods also bring along more knowledge of the model.”
MethodGurina, E., Klyuchnikov, N., Antipova, K., & Koroteev, D. A. (2022) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2022.111041 · Cited by 18
“In the following paragraphs, we briefly describe the state-of-the-art solutions, while the complete comparison of the presented methods can be found in papers (ElShawi et al., 2021; Carvalho et al., 2019).”
BackgroundMethodHuman-AI Collaboration: Exploring interfaces for interactive Machine Learning
al., G. C. S. E. (2023) · Journal of Propulsion Technology · doi:10.52783/tjjpt.v44.i2.148 · Cited by 31
Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability
Mikriukov, G., Schwalbe, G., Hellert, C., & Bade, K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_26 · Cited by 20
“XAI is a subfield of AI that focuses on revealing the inner workings of blackbox models in a way that humans can understand [37,5,27].”
BackgroundA Comparative Analysis of Feature Eliminator Methods to Improve Machine Learning Phishing Detection
Tanimu, J., Shiaeles, S. N., & Adda, M. (2023) · Journal of Data Science and Intelligent Systems · doi:10.47852/bonviewjdsis32021736 · Cited by 18
Top three intelligent algorithms for OAM mode recognitions in optical communications
Wang, B., Zhang, X., Shah, S. A. A., Merabet, B., Kovalev, A. A., Stafeev, S. S., Kozlova, E. S., Kotlyar, V. V., & Guo, Z. (2024) · Engineering Research Express · doi:10.1088/2631-8695/ad61bc · Cited by 17
Finzel, B. (2025) · Pflügers Archiv - European Journal of Physiology · doi:10.1007/s00424-025-03067-7 · Cited by 21
“The interested reader may find more information in state-of-the-art survey papers on XAI methods [15, 74, 75, 115, 131].”
Explainable Supervised Learning Models for Aviation Predictions in Australia
Nanyonga, A., Wasswa, H., Joiner, K. F., Turhan, U., & Wild, G. (2025) · Aerospace · doi:10.3390/aerospace12030223 · Cited by 17
Token-wise sentiment decomposition for ConvNet: Visualizing a sentiment classifier
Chawla, P., Hazarika, S., & Shen, H. (2020) · Visual Informatics · doi:10.1016/j.visinf.2020.04.006 · Cited by 22
Neural Image Compression and Explanation
Li, X., & Ji, S. (2020) · IEEE Access · doi:10.1109/access.2020.3041416 · Cited by 20
“These methods can be categorized based on whether it is designed to explain the entire model behavior (global interpretability) or a single prediction (local interpretability) [25].”
BackgroundJain, N., & Jana, P. K. (2022) · Information Sciences · doi:10.1016/j.ins.2022.09.040 · Cited by 21
FFTree: A flexible tree to handle multiple fairness criteria
Castelnovo, A., Cosentini, A., Malandri, L., Mercorio, F., & Mezzanzanica, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103099 · Cited by 22
Black-box error diagnosis in Deep Neural Networks for computer vision: a survey of tools
Fraternali, P., Milani, F., Torres, R. N., & Zangrando, N. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08100-9 · Cited by 18
Dong, Z., Zhang, H., Chen, Y., Payne, P., & Li, F. (2023) · Cancers · doi:10.3390/cancers15174210 · Cited by 20
“Tran-Synergy [21] applies the post hoc interpretation framework [24] that computes the Shapley value of each gene through GradientExplainer and uses DeepExplainer to characterize its contribution to the final synergy prediction.”
MethodMachine learning modeling for predicting adherence to physical activity guideline
Choe, J.-P., Lee, S., & Kang, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-90077-1 · Cited by 19
“To address this issue, studies about feature importance have been widely progressed by researchers to interpret the result of ML 40,41 .”
Borde, H. S. D. O., Sondak, D., & Protopapas, P. (2021) · Journal of Turbulence · doi:10.1080/14685248.2021.1999459 · Cited by 22
SF-CNN: Signal Filtering Convolutional Neural Network for Precipitation Intensity Estimation
Lin, C.-W., Huang, X., Lin, M., & Hong, S.-D. (2022) · Sensors · doi:10.3390/s22020551 · Cited by 16
“In other words, it is difficult to explain its working mechanism for specific reasons for the formation of the features and decision boundaries in the form of mathematical expressions [62].”
BackgroundWang, S., Mei, Y., & Zhang, M. (2023) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2023.3238741 · Cited by 19
“Transparency refers to the degree that a model can be understood by users directly [12].”
BackgroundMethodInfluentialMugisha, C., & Paik, I. (2023) · Electronics · doi:10.3390/electronics12081848 · Cited by 17
Zhang, Y., Golbus, J. R., Wittrup, E. M., Aaronson, K. D., & Najarian, K. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02453-y · Cited by 18
“…issue of opacity and lack of justification for decision-making which has hindered their applications in medicine [15], where model interpretability allows clinicians to comprehend the rationale behind the model’s predictions, thereby facilitating the identification of new risk factors [16].”
BackgroundAn Empirical Evaluation of Shapley Additive Explanations: A Military Implication
Rao, T., Agarwal, S., & Singh, N. (2023) · IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON) · doi:10.1109/upcon59197.2023.10434608 · Cited by 16
“The authors of [19] investigated three different agnostic interpretable approaches (Anchors, SHAP, LIME) on diverse medical data with a stronger healthcare-related focus.”
BackgroundBalancing XAI with Privacy and Security Considerations
Spartalis, C. N., Semertzidis, T., & Daras, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54129-2_7 · Cited by 24
“Methods applied after model training to clarify model decisions; typically associated with model-agnostic methods [8].”
BackgroundMethodInfluentialEllaky, Z., & Benabbou, F. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02269 · Cited by 18
Bouasabah, M. (2024) · Economies · doi:10.3390/economies12080194 · Cited by 16
“These metrics are computed for each model and subsequently compared to comprehensively assess their predictive capabilities Carvalho et al. (2019).”
MethodLi, W., Wang, X., Li, G., Geng, B., & Varshney, P. K. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3524639 · Cited by 17
“Here, we introduce the taxonomy of interpretable ML methods used in [39] and [41], where interpretability in ML methods is classified into three categories, that is, pre-model interpretability, in-model interpretability, and post-model interpretability.”
InfluentialGai, Y., & Liu, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107504 · Cited by 16
A Review of Interpretability Methods for Gradient Boosting Decision Trees
Gonçalves, V. S. F., & de Carvalho, V. R. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5324 · Cited by 23
“…are versatile and can be used to explain a wide range of models, whereas model-specific methods are designed for particular types of models [Alex Goldstein and Pitkin, 2015] [Rawal et al ., 2022] [Islam et al ., 2021] [Das et al ., 2020] [Carvalho et al ., 2019] [Nagahisarchoghaei et al ., 2023].”
Rahman, M. M., Calhoun, V. D., & Plis, S. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1129 · Cited by 16
Leveraging online behaviors for interpretable knowledge-aware patent recommendation
Du, W., Yan, Q., Zhang, W., & Ma, J. (2021) · Internet Research · doi:10.1108/intr-08-2020-0473 · Cited by 16
Interpretable Machine Learning: A Case Study of Healthcare
Okay, F. Y., Yildirim, M., & Özdemir, S. (2021) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc52172.2021.9615727 · Cited by 21
How to Find a Good Explanation for Clustering?
Bandyapadhyay, S., Fomin, F. V., Golovach, P. A., Lochet, W., Purohit, N., & Simonov, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20306 · Cited by 25
An ensemble meta-estimator to predict source code testability
Nasrabadi, M. Z., & Parsa, S. A. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109562 · Cited by 18
“To support the interpretability of the learned model [7], we applied a feature importance analysis technique [8], computing the impact of each metric on testability prediction.”
MethodBusiness Valuation with Machine learning
Коклев, П. С. (2022) · Finance: Theory and Practice · doi:10.26794/2587-5671-2022-26-5-132-148 · Cited by 22
Transforming Regulatory Reporting with AI/ML: Strategies for Compliance and Efficiency
Tillu, R., -, M. M., & Periyasamy, V. (2023) · Journal of Knowledge Learning and Science Technology · doi:10.60087/jklst.vol2.n1.p157 · Cited by 16
Clemens, T., & Viechtbauer-Gruber, M. (2020) · SPE Reservoir Evaluation & Engineering · doi:10.2118/200540-pa · Cited by 17
Chan, G. Y.-Y., Bertini, E., Nonato, L. G., Barr, B., & Silva, C. T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10614 · Cited by 20
“In detail, the tasks in the above workflow generalize the studies that consolidate the key user requirements of model explainability [1,6,14, 16,37,44].”
BackgroundMapping the Potential of Explainable AI for Fairness Along the AI LifecyclePreprint
Deck, L., Schomäcker, A., Speith, T., Schöffer, J., Kästner, L., & Kühl, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.18736 · Cited by 23
“If an unfavorable decision was based on (a proxy for) the skin color of a person, this person should supposedly be able to recognize that they were treated unfairly [22, 77].”
Brunet-Gouet, É., Vidal, N., & Roux, P. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7637475 · Cited by 14
Plant disease epidemiology in the age of artificial intelligence and machine learning
Neik, T. X., Dolatabadian, A., Danilevicz, M. F., Upadhyaya, S. R., Zhang, F., Batley, J., & Edwards, D. (2025) · Agriculture Communications · doi:10.1016/j.agrcom.2025.100089 · Cited by 26
A Multi-Criteria Decision-Making Approach for the Selection of Explainable AI Methods
Matejová, M., & Paralič, J. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7040158 · Cited by 17
“Various and significantly different methods have been developed in the field of ex-plainability [2,12,13].”
Why Anomaly-Based Intrusion Detection Systems Have Not Yet Conquered the Industrial Market?
Seng, S., García-Alfaro, J., & Laarouchi, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08147-7_23 · Cited by 18
“[32] identified at least 14 workshops or symposia dedicated to this thematic between 2014 and mid-2019.”
BackgroundA systematic literature review on AutoML for multi-target learning tasks
Valle, A. M. D., Mantovani, R. G., & Cerri, R. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10569-2 · Cited by 15
“For example, as for the scope, the technique can be local or global, and as for the taxonomy, model-specific or model-agnostic (Carvalho et al. 2019”
MethodAlJalaud, E., & Hosny, M. (2024) · IEEE Access · doi:10.1109/access.2024.3404043 · Cited by 17
“It is described as local because the prediction may depend linearly on some features rather than having a complex dependence on them [12].”
BackgroundZhou, Y., Zhang, X., Gong, J., Wang, T., Gong, L., Li, K., & Wang, Y. (2024) · Journal of Adolescence · doi:10.1002/jad.12357 · Cited by 17
Staining to machine learning: An emerging technology for determination of microalgal cell viability
Kim, T., Pradhan, B., & Ki, J. (2024) · Journal of Applied Phycology · doi:10.1007/s10811-024-03274-2 · Cited by 16
“It requires input data be prepared in the same way; otherwise, the analysis's quality may suffer (Ribeiro et al. 2016; Shirazi et al. 2018; Carvalho et al. 2019).”
RM2D: An automated and robust laser-based framework for mobile tunnel deformation detection
Chen, B., Zhao, Z., Bi, L., & Wang, Z. (2024) · Underground Space · doi:10.1016/j.undsp.2024.07.002 · Cited by 15
Velmurugan, M., Ouyang, C., Xu, Y., Sindhgatta, R., Wickramanayake, B., & Moreira, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109772 · Cited by 18
Exploring the means to measure explainability: Metrics, heuristics and questionnaires
Deters, H., Droste, J., Obaidi, M., & Schneider, K. A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107682 · Cited by 19
A method for unsupervised learning of coherent spatiotemporal patterns in multiscale data
Lapo, K., Ichinaga, S. M., & Kutz, J. N. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415786122 · Cited by 20
Application of metaheuristic spotted hyena optimization in strength prediction of concrete
Parhi, S. K., & Panigrahi, S. K. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29162-3.00008-3 · Cited by 15
Lin, L., Walker, C. M., & Agarwal, V. (2025) · Nuclear Engineering and Technology · doi:10.1016/j.net.2025.103588 · Cited by 19
Testing the Skill of a Species Distribution Model Using a 21st Century Virtual Ecosystem
Bardon, L. R., Ward, B. A., Dutkiewicz, S., & Cael, B. B. (2021) · Geophysical Research Letters · doi:10.1029/2021gl093455 · Cited by 16
Tutul, A. A., Nirjhar, E. H., & Chaspari, T. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479926 · Cited by 16
“The performance and characteristics of the machine learning (ML) algorithm that powers the AI system, including the transparency, explainability, privacy preservation, and fairness, have been posited as important system-based factors [4, 18, 31].”
BackgroundAn explanatory analytics framework for early detection of chronic risk factors in pandemics
Davazdahemami, B., Zolbanin, H. M., & Delen, D. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100020 · Cited by 16
“In recent years, however, there has been a stream of research roposing various approaches, such as LIME [25], DeepLIFT [26], and ayer-Wise Relevance Propagation [27], to improve the interpretability spect of ML (See [28,29] for a comprehensive review of such methds).”
Aslam, N. (2022) · Computation · doi:10.3390/computation10030036 · Cited by 20
“Nevertheless, ML and DL have high predictive power but lack interpretability [10].”
BackgroundML-Based Classification of Device Environment Using Wi-Fi and Cellular Signal Measurements
Ramamurthy, A., Sathya, V., Rochman, M. I., & Ghosh, M. (2022) · IEEE Access · doi:10.1109/access.2022.3158056 · Cited by 15
MUBot: Learning to Test Large-Scale Commercial Android Apps like a Human
Peng, C., Zhang, Z., Lv, Z., & Yang, P. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00074 · Cited by 23
An Explainable Machine Learning Approach to Predicting and Understanding Dropouts in MOOCs
Er, E. (2023) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1246458 · Cited by 17
“An explainable machine learning model differs from a traditional model in that a human can interpret and judge why predictions were made in a certain way (Carvalho et al., 2019).”
Taborri, J., Palermo, E., & Rossi, S. (2023) · Sensors · doi:10.3390/s23115245 · Cited by 15
“This interesting outcome permits the affirmation of the robustness of the implemented algorithms to both type I and type II errors [33].”
BackgroundGonzález‐González, J., de Arriba-Pérez, F., García-Méndez, S., Busto-Castiñeira, A., & González-Castaño, F. J. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101634 · Cited by 18
“In this regard, safety and interpretability (Carvalho et al., 2019; Linardatos et al., 2020) pursue the extraction of knowledge from the trained classifiers by traversing the decision tree paths to better understand the performance of the ML model.”
IEC 61499 Device Management Model through the lenses of RMAS
Bonci, A., Longhi, S., & Pirani, M. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.01.288 · Cited by 16
Perera, B. S., García-Barceló, C., Arcas, M. G., & Gil, D. (2025) · Information · doi:10.3390/info16090763 · Cited by 14
“This type of analysis is particularly useful for identifying consistent trends and informing high-level educational policy decisions [14].”
Contributions of Artificial Intelligence to Decision Making in Nursing: A Scoping Review
Fernandes, F., Shinners, L., Mota, M., Santos, P., & Sá, L. (2026) · Nursing and Health Sciences · doi:10.1111/nhs.70308 · Cited by 17
“…in adopting AI technologies is ensuring that the decision-making processes of these models are interpretable and transparent, allowing nurses and other healthcare professionals to clearly understand how AI-driven decisions are made, mainly when patient safety is a concern (Carvalho et al. 2019).”
Explainable AI for Medical Imaging: Knowledge Matters
Bourdon, P., Ahmed, O. B., Urruty, T., Djemal, K., & Fernández-Maloigne, C. (2012) · Springer eBooks · doi:10.1007/978-3-030-74478-6_11 · Cited by 14
Linder, R., Hunter, C., McLemore, J., Dutta, S., Akbar, F., Grover, T., Breideband, T., Borghouts, J. W., Lu, Y., Mark, G. J., et al. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492859 · Cited by 16
“These observations highlight the necessity for advancements in AI methods that facilitate autonomous systems with amplified interpretability [30] and human-in-the-loop systems that rely on guidance from an extremely small number of qualified people [35].”
BackgroundAI-assisted Synthesis in Next Generation EDA: Promises, Challenges, and Prospects
Wu, N., Xie, Y., & Hao, C. (2022) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd56317.2022.00039 · Cited by 18
“Thus, efforts in interpretable ML [61], [62] are highly expected to promote production-ready applications of ML for EDA.”
BackgroundNascimben, M., Lippi, L., Alessandro de, S. I. R. E., Invernizzi, M., & Rimondini, L. (2023) · Cancers · doi:10.3390/cancers15020336 · Cited by 14
“Indeed, one criticism connected with machine learning in medicine is the lack of appealing explanations of artificial intelligence models [67].”
BackgroundA state‐of‐the‐art survey of predicting students' performance using artificial neural networks
Xiao, W. J., & Hu, J. (2023) · Engineering Reports · doi:10.1002/eng2.12652 · Cited by 14
AraFake: A Deep Learning Approach for Arabic Fake News Detection
AbdElminaam, D. S., Abdelaziz, A., Essam, G., & Mohamed, S. E. (2023) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc58832.2023.10278341 · Cited by 16
Banana Ripeness Classification with Deep CNN on NVIDIA Jetson Xavier AGX
Aishwarya, N., & Vinesh Kumar, R. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290326 · Cited by 14
“Additionally, ML offers a toolkit of strong algorithms [10, 11] that enable effective classification performance.”
BackgroundZhang, Y., Chen, F., Liu, D., Wang, J., Feng, Q., Jiang, H., Wang, X., Zhao, H., Zhong, S., Shah, F. M., et al. (2023) · IET Nanodielectrics · doi:10.1049/nde2.12071 · Cited by 14
Zhao, H., Jiming, E., Chen, S., Cheng, G., & Guo, F. (2023) · Tribology International · doi:10.1016/j.triboint.2023.109184 · Cited by 14
Li, Z., Bouazizi, M., Ohtsuki, T. O., Ishii, M., & Nakahara, E. (2023) · IEEE Access · doi:10.1109/access.2023.3347796 · Cited by 16
“According to Carvalho et al., there are mainly three goals for better explainability: (1) user’s understandability, (2) user’s interpreting efficiency, and (3) XAI approximating accuracy [7].”
BackgroundAdaptable multi-objective optimization framework: application to metal additive manufacturing
Heddar, M. I. E., Mehdi, B., Matougui, N., Tahan, S. A., & Jahazi, M. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-13489-9 · Cited by 15
“While surrogate modeling offers a computationally efficient alternative to direct optimization, traditional surrogate-based approaches often suffer from a lack of interpretability [30, 32, 33].”
Mpox-AISM: AI-mediated super monitoring for mpox and like-mpox
Yue, Y., Li, Z., Zhang, X., Wang, Z., Zhang, X., Zhang, F., Saravanakumar, S., Xu, J., Zhao, W., Liu, J., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109766 · Cited by 14
“The lack of interpretability raises a severe issue about the trust of deep models in high-stakes prediction applications, such as autonomous driving, healthcare, criminal justice, and financial services [16].”
BackgroundMachine learning models’ assessment: trust and performance
Sousa, S., Paredes, S., Rocha, T., Henriques, J. M. O., Sousa, J. P., & Gonçalves, L. M. M. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03145-5 · Cited by 15
“Adapting the taxonomy adopted in [8], besides the proposed quantitative assessment of trust (commonly designated as functionally grounded evaluation), a qualitative assessment performed by domain experts can also be accomplished (commonly designated as application-grounded evaluation).”
MethodMoise, G., Dragomir, E. G., Șchiopu, D., & Iancu, L. (2024) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-024-00638-x · Cited by 14
“A comprehensive study on methods and metrics used in the field of machine-learning interpretability can be found in [54].”
Stoykova, R., Porter, K., & Beka, T. (2024) · Forensic Science International Synergy · doi:10.1016/j.fsisyn.2024.100563 · Cited by 16
Gómez-Talal, I., Azizsoltani, M., Bote-Curiel, L., Rojo‐Álvarez, J. L., & Singh, A. K. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110664 · Cited by 14
AI-powered optimization and numerical techniques for nanofluid heat transfer systems-a review
Raza, M., Faiz, M., Hassan, W. U. I., Abbas, M. S., Raza, J., Kumail, Z., Nawaz, T., Shabir, S., Jan, A., & Li, F.-C. (2025) · Multiscale and Multidisciplinary Modeling Experiments and Design · doi:10.1007/s41939-025-00891-3 · Cited by 17
“…predictions restricts their practical applicative usability and complicates trust between researchers and industrial practitioners in (engineering industrial) applications where reliability and explainability are paramount (Nguyen et al. 2019; Carvalho et al. 2019; Jamei and Said 2022).”
Explainable AI and machine learning for robust cybersecurity in smart cities
Gupta, S., Singh, J., Agrawal, R., & Batra, U. (2025) · Cyber Security and Applications · doi:10.1016/j.csa.2025.100104 · Cited by 18
Chiarelli, A. M., Sestieri, C., Navarra, R., Wise, R. G., & Caulo, M. (2021) · Human Brain Mapping · doi:10.1002/hbm.25456 · Cited by 14
Christoforou, A. P., Andreou, A. S., Garriga, M., & Baresi, L. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108066 · Cited by 15
Agroecosystem engineering extended from plant-microbe interactions revealed by multi-omics data
Fujiwara, F., Miyazawa, K., Nihei, N., & Ichihashi, Y. (2022) · Bioscience Biotechnology and Biochemistry · doi:10.1093/bbb/zbac191 · Cited by 15
MetaDT: Meta Decision Tree With Class Hierarchy for Interpretable Few-Shot Learning
Zhang, B., Jiang, H., Li, X., Feng, S., Ye, Y., Luo, C., & Ye, R. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3227574 · Cited by 16
From Natural Language to Workflows: Towards Emergent Intelligence in Robotic Process Automation
Chakraborti, T., Rizk, Y., Isahagian, V., Aksar, B., & Fuggitti, F. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-16168-1_8 · Cited by 16
Zhuparris, A., de Goede, A. A., Yocarini, I. E., Kraaij, W., Groeneveld, G. J., & Doll, R. J. (2023) · Sensors · doi:10.3390/s23115243 · Cited by 16
Du, Y., Liu, Y., Yan, Y., Fang, J., & Jiang, X. (2023) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2022.000430 · Cited by 15
Dissecting users' needs for search result explanations
Juneja, P., Zhang, W., Smith, A., Lamba, H., Tetreault, J., & Jaimes, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642059 · Cited by 17
“…determining users’ needs from explanations [47–49], determining the impact of explanations on user perceptions of and resulting interactions with these systems [10, 13, 13, 16, 17, 41, 42, 44, 49, 66], as well as developing metrics to evaluate the quality and effectiveness of explanations [12, 33].”
BackgroundTowards Incorporating Human Knowledge in Fuzzy Pattern Tree Evolution
Murphy, A., Murphy, G., Amaral, J. L. M., Dias, D. M., Naredo, E., & Ryan, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72812-0_5 · Cited by 14
Metadata
- Authors
- Carvalho, D. V., Pereira, E. M., & Cardoso, J. S.
- Affiliations
- Deloitte Portugal, Manuel Bandeira Street, 43, 4150-479 Porto, Portugal
- Deloitte Portugal (Portugal) (PT)
- Universidade do Porto (PT)
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2019-07-26
- Volume
- 8
- Issue
- 8
- Pages
- 832
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W2964303497
- Cited by
- 1,842
- References
- 106
- Citations by year
- 2026: 224 · 2025: 354 · 2024: 353 · 2023: 313 · 2022: 285 · 2021: 218 · 2020: 91 · 2019: 3
- FWCI
- 68.32
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- model interpretability, explainable AI, explanation quality metrics, interpretable models, explanation methods, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.