The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Adversarial attacks and defenses in explainable artificial intelligence: A survey
Open accessAbstract
Explainable artificial intelligence (XAI) methods are portrayed as a remedy for debugging and trusting statistical and deep learning models, as well as interpreting their predictions. However, recent advances in adversarial machine learning (AdvML) highlight the limitations and vulnerabilities of state-of-the-art explanation methods, putting their security and trustworthiness into question. The possibility of manipulating, fooling or fairwashing evidence of the model's reasoning has detrimental consequences when applied in high-stakes decision-making and knowledge discovery. This survey provides a comprehensive overview of research concerning adversarial attacks on explanations of machine learning models, as well as fairness metrics. We introduce a unified notation and taxonomy of methods facilitating a common ground for researchers and practitioners from the intersecting research fields of AdvML and XAI. We discuss how to defend against attacks and design robust interpretation methods. We contribute a list of existing insecurities in XAI and outline the emerging research directions in adversarial XAI (AdvXAI). Future work should address improving explanation methods and evaluation protocols to take into account the reported safety issues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
One Pixel Attack for Fooling Deep Neural Networks
Su, J., Vargas, D. V., & Sakurai, K. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2890858 · Cited by 2,406
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“• Anchors: high-precision model-agnostic explanations [Ribeiro et al., 2018]”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · cited 1×
“Currently, there is an open discussion on whether the developed XAI methodology is su ffi cient to attain trustworthy AI systems [74, 75].”
BackgroundSmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
Evasion Attacks against Machine Learning at Test Time
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_25 · Cited by 890 · cited 1×
“Figure 1 illustrates common failure modes that exist due to data and models influencing the outputs of explanation methods.”
BackgroundInterpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · cited 1×
“We purposely exclude a large number of papers focusing primarily on explanation evaluation without relating to the adversarial scenario [refer to 50, 51].”
BackgroundNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Brown, T. B., Mané, D., Roy, A., Abadi, M., & Gilmer, J. (2017) · arXiv · doi:10.48550/arxiv.1712.09665 · Cited by 509
SoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 330
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295
Stronger Data Poisoning Attacks Break Data Sanitization Defenses
Koh, P. W., Steinhardt, J., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1811.00741 · Cited by 189
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171
The European legislation on AI: A brief analysis of its philosophical approach
Floridi, L. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00460-9 · Cited by 169
Acquisition of chess knowledge in AlphaZero
McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U., & Kramnik, V. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2206625119 · Cited by 166
Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
Cinà, A. E., Grosse, K., Demontis, A., Vascon, S., Zellinger, W., Moser, B. A., Oprea, A., Biggio, B., Pelillo, M., & Roli, F. (2023) · ACM Computing Surveys · doi:10.1145/3585385 · Cited by 152
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125
Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security
Kuppa, A., & Le-Khac, N.-A. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206780 · Cited by 123
Hedström, A., Weber, L., Bareeva, D., Krakowczyk, D., Motzkus, F., Samek, W., Lapuschkin, S., & Höhne, M. M. C. (2022) · arXiv · doi:10.48550/arxiv.2202.06861 · Cited by 116
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54
Watkins, E. A., McKenna, M., & Chen, J. (2022) · arXiv · doi:10.48550/arxiv.2202.09519 · Cited by 35
Watkins, E. A., & Chen, J. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658938 · Cited by 25
Cited by
Artificial intelligence in environmental monitoring: Advancements, challenges, and future directions
Olawade, D. B., Wada, O. Z., Ige, A. O., Egbewole, B. I., Olojo, A., & Oladapo, B. I. (2024) · Hygiene and Environmental Health Advances · doi:10.1016/j.heha.2024.100114 · Cited by 326
Xin, X., Hooker, G., & Huang, F. (2025) · Insurance: Mathematics and Economics · doi:10.1016/j.insmatheco.2025.103135 · Cited by 6
Metadata
- Authors
- Baniecki, H., & Biecek, P.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2024-07
- Volume
- 107
- Pages
- 102303
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W4391933803
- Cited by
- 170
- References
- 42
- FWCI
- 24.70
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.