CoChat
Sign Up
Cited by 124Explainable Artificial Intelligence (XAI)Software Testing and Debugging TechniquesAdversarial Robustness in Machine Learning

The DOI matches its Crossref record; 245 of the 245 references it cites were found in a registry or an index.

EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications

Open access
Ortigossa, Evandro S., Gonçalves, Thales, Nonato, Luis Gustavo·2024·IEEE Access
DOI 10.1109/access.2024.3409843
Open this paper in CoChat — free account

Abstract

Intelligent applications supported by Machine Learning have achieved remarkable performance rates for a wide range of tasks in many domains. However, understanding why a trained algorithm makes a particular decision remains problematic. Given the growing interest in the application of learning-based models, some concerns arise in the dealing with sensible environments, which may impact users’ lives. The complex nature of those models’ decision mechanisms makes them the so-called “black boxes,” in which the understanding of the logic behind automated decision-making processes by humans is not trivial. Furthermore, the reasoning that leads a model to provide a specific prediction can be more important than performance metrics, which introduces a trade-off between interpretability and model accuracy. Explaining intelligent computer decisions can be regarded as a way to justify their reliability and establish trust. In this sense, explanations are critical tools that verify predictions to discover errors and biases previously hidden within the models’ complex structures, opening up vast possibilities for more responsible applications. In this review, we provide theoretical foundations of Explainable Artificial Intelligence (XAI), clarifying diffuse definitions and identifying research objectives, challenges, and future research lines related to turning opaque machine learning outputs into more transparent decisions. We also present a careful overview of the state-of-the-art explainability approaches, with a particular analysis of methods based on feature importance, such as the well-known LIME and SHAP. As a result, we highlight practical applications of the successful use of XAI.

Overview

Integrity signals

Registry

Crossref · 10.1109/access.2024.3409843

Cites

245 of 245 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 52

  1. 2023, 2016/679 of the European Parliament and of the council of 27 April 2016 on the General Data Protection Regulation
  2. 2024, Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)—Discussion Paper and Request for Feedback
  3. 10.1016/j.media.2021.102306 · DOI
  4. 2021, Ethics and Governance of Artificial Intelligence for Health: WHO Guidance
  5. 2021, California Consumer Privacy Act (CCPA)
  6. 10.3389/fdata.2021.688969 · DOI · Paper page
  7. 2016, The Neural Network Zoo—The Asimov Institute
  8. Ghojogh, 2019, The theory behind overfitting, cross validation, regularization, bagging, and boosting: Tutorial, arXiv:1905.12787 · arXiv
  9. 10.48550/ARXIV.1706.03762 · DOI · Paper page
  10. Neal, 2018, A modern take on the bias-variance tradeoff in neural networks, arXiv:1810.08591 · arXiv
  11. Goodfellow, 2016, Deep Learning
  12. Wang, Benign overfitting in multiclass classification: All roads lead to interpolation, 34, 24164
  13. 10.2307/2699986 · DOI
  14. Kumar, Shapley residuals: Quantifying the limits of the Shapley value for explanations, 34, 1
  15. Vashishth, 2019, Attention interpretability across NLP tasks, arXiv:1909.11218 · arXiv
  16. Munroe, 2023, XKCD—A Webcomic of Romance, Sarcasm, Math, and Language
  17. Yousefzadeh, 2019, Interpreting neural networks using flip points, arXiv:1903.08789 · arXiv
  18. Ba, Do deep nets really need to be deep?, 27, 1
  19. Kass, 1988, The need for user models in generating expert system explanations, Int. J. Exp. Syst., 1, 1
  20. Kumar, 2021, Is Shapley explanation for a model unique?, arXiv:2111.11946 · arXiv
  21. Wojtas, Feature importance ranking for deep learning, 33, 5105
  22. Alvarez-Melis, 2018, On the robustness of interpretability methods, arXiv:1806.08049 · arXiv
  23. Mishra, 2021, A survey on the robustness of feature importance and counterfactual explanations, arXiv:2111.00358 · arXiv
  24. Guidotti, 2018, Local rule-based explanations of black box decision systems, arXiv:1805.10820 · arXiv
  25. Xenopoulos, GALE: Globally assessing local explanations, 322
  26. 10.2139/ssrn.3063289 · DOI · Paper page
  27. Ying, GNNexplainer: Generating explanations for graph neural networks, 32, 1
  28. Vu, PGM-Explainer: Probabilistic graphical model explanations for graph neural networks, 33, 12225
  29. Yuan, On explainability of graph neural networks via subgraph explorations, 12241
  30. Baldassarre, 2019, Explainability techniques for graph convolutional networks, arXiv:1905.13686 · arXiv
  31. Biderman, Pythia: A suite for analyzing large language models across training and scaling, 2397
  32. Vig, BertViz: A tool for visualizing multihead self-attention in the BERT model, 23, 1
  33. Sundararajan, Axiomatic attribution for deep networks, 70, 3319
  34. Hamilton, 2021, Axiomatic explanations for visual search, retrieval, and similarity learning, arXiv:2103.00370 · arXiv
  35. Smilkov, 2017, SmoothGrad: Removing noise by adding noise, arXiv:1706.03825 · arXiv
  36. Shrikumar, Learning important features through propagating activation differences, 3145
  37. Sundararajan, The many Shapley values for model explanation, 9269
  38. Yousefzadeh, 2019, Investigating decision boundaries of trained neural networks, arXiv:1908.02802 · arXiv
  39. Lakkaraju, Robust and stable black box explanations, 5628
  40. Lundberg, 2018, Consistent individualized feature attribution for tree ensembles, arXiv:1802.03888 · arXiv
  41. 10.1007/s11222-021-10057-z · DOI · Paper page
  42. Barr, The disagreement problem in faithfulness metrics, 1
  43. Ye, Variable selection via penalized neural network: A drop-out-one loss approach, 5620
  44. Weerts, 2019, A human-grounded evaluation of SHAP for alert processing, arXiv:1907.03324 · arXiv
  45. Tsirtsis, Decisions, counterfactual explanations and strategic behavior, 33, 16749
  46. Alvarez-Melis, 2018, Towards robust interpretability with self-explaining neural networks, arXiv:1806.07538 · arXiv
  47. Petsiuk, 2018, RISE: Randomized input sampling for explanation of black-box models, arXiv:1806.07421 · arXiv
  48. Agarwal, 2022, Rethinking stability for attribution-based explanations, arXiv:2203.06877 · arXiv
  49. Han, Which explanation should I choose? A function approximation perspective to characterizing post hoc explanations, 35, 5256
  50. Banachewicz, 2022, The Kaggle Book: Data Analysis and Machine Learning for Competitive Data Science
  51. Alber, 2019, iNNvestigate neural networks!, J. Mach. Learn. Res., 20, 1
  52. Preece, 2018, Stakeholders in explainable AI, arXiv:1810.00184 · arXiv

Cited by

Sorted by citations

See all 124

Related

See all

Metadata

Authors
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G.
Affiliations
  • Universidade de São Paulo (BR)
  • Institute of Mathematics and Computer Science, University of São Paulo (ICMC-USP), São Carlos, Brazil
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024
Volume
12
Pages
80799-80846
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2024.3409843
OpenAlex
W4399449888
Cited by
124
References
245
Citations by year
2026: 66 · 2025: 52 · 2024: 6
FWCI
18.26
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior–Brasil (CAPES)–Finance Code 001
  • São Paulo Research Foundation
  • National Council for Scientific and Technological Development
  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
  • Fundação de Amparo à Pesquisa do Estado de São Paulo (2018/24516-0, 2022/09091-8)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (307184/2021-8)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
explainable AI, model interpretability, black-box models, automated decision-making, trust in machine learning, feature importance, lime, SHAP, model transparency, algorithmic bias detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review