Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
Cybersecurity community is slowly leveraging Machine Learning (ML) to combat ever evolving threats. One of the biggest drivers for successful adoption of these models is how well domain experts and users are able to understand and trust their functionality. As these black-box models are being employed to make important predictions, the demand for transparency and explainability is increasing from the stakeholders.Explanations supporting the output of ML models are crucial in cyber security, where experts require far more information from the model than a simple binary output for their analysis. Recent approaches in the literature have focused on three different areas: (a) creating and improving explainability methods which help users better understand the internal workings of ML models and their outputs; (b) attacks on interpreters in white box setting; (c) defining the exact properties and metrics of the explanations generated by models. However, they have not covered, the security properties and threat models relevant to cybersecurity domain, and attacks on explainable models in black box settings.In this paper, we bridge this gap by proposing a taxonomy for Explainable Artificial Intelligence (XAI) methods, covering various security properties and threat models relevant to cyber security domain. We design a novel black box attack for analyzing the consistency, correctness and confidence security properties of gradient based XAI methods. We validate our proposed system on 3…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This supports previous claims that explainability is both a technical and regulatory requirement under frameworks like GDPR [56].”
From Application of Machine Learning and Deep Learning Techniques for Enhanced Insider Threat Detection in Cybersecurity: Bibliometric Review (2025) · cites this paper as [56]
In the citing paper
“However, most XAI implementations remain post hoc rather than intrinsic, confirming the research gap also noted in recent surveys [41,56].”
From Application of Machine Learning and Deep Learning Techniques for Enhanced Insider Threat Detection in Cybersecurity: Bibliometric Review (2025) · cites this paper as [56]
Citations
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“In [30] showed that the post-hoc explanations are not faithful but only present correlations of the underlying computations.”
BackgroundRobust Statistics: The Approach Based on Influence Functions
Ruppert, D. (1987) · Technometrics · doi:10.2307/1269782 · Cited by 3,880
Drebin: Effective and Explainable Detection of Android Malware in Your Pocket
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014) · Proceedings 2014 Network and Distributed System Security Symposium · doi:10.14722/ndss.2014.23247 · Cited by 2,331
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“Not being able to understand what has been captured by the model [6] and stored in its internal representation may entail a privacy breach.”
BackgroundSurvey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Interpretation of Neural Networks is FragilePreprint
Ghorbani, A., Abid, A., & Zou, J. Y. (2017) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1710.10547 · Cited by 1,153
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“[13] showed that in the case of structured data, LIME and SHAP explanations are not intuitive.”
BackgroundInterpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Adversarial Examples for Malware Detection
Grosse, K., Papernot, N., Manoharan, P., Backes, M., & McDaniel, P. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66399-9_4 · Cited by 542
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404
Research works printed but not linked: 8
- Swartout, 1983, XPLAIN: A system for creating and explaining expert consulting programs Artificial intelligence
- 10.1145/279232.279236 · DOI · Paper page
- Jones, 2014, {SciPy}: Open source scientific tools for {Python}
- 10.1111/rssb.12508 · DOI
- 10.1080/00401706.1987.10488218 · doi:10.1080/00401706.1987.10488218 · Paper page
- 10.1109/eurospw51379.2020.00045 · DOI
- Hall, A Systematic Method to Understand Requirements for Explainable AI (XAI) Systems
- 10.1609/aaai.v33i01.33013681 · DOI · Paper page
Cited by
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
“Several works have advocated the idea of exploiting explanations beyond just explaining decisions [30,167], for instance, also to improve models.”
BackgroundExplainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
Zhang, Z., Hamadi, H. A., Damiani, E., Yeun, C. Y., & Taher, F. (2022) · IEEE Access · doi:10.1109/access.2022.3204051 · Cited by 450
“[260] to examine the consistency, accuracy, and confidence security characteristics of gradient-based XAI algorithms.”
BackgroundExplainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295
“different black-box attacks were attempted by the adversary: an attack based on zeroth order optimization (ZOO; see Section 3), an attack based on a GAN (similar to the one proposed by Lin et al. (2018)), and an attack on which a substitute model is trained which is followed by a C&W attack (see Section 3) performed a…”
MethodInfluential
Metadata
- Authors
- Kuppa, A., & Le-Khac, N.-A.
- Affiliations
- University College Dublin (IE)
- ORCID
- Kuppa, Aditya
- Venue
- 2020 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2020-07
- Pages
- 1-8
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W3091163785
- Semantic Scholar
- 8ad869232616f72d5cfe5b905d52862ef639743f
- Cited by
- 123
- References
- 35
- Citations by year
- 2026: 13 · 2025: 24 · 2024: 32 · 2023: 25 · 2022: 17 · 2021: 11 · 2020: 1
- FWCI
- 5.15
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- explainable AI, black-box attacks, threat model, gradient-based explanation, explanation consistency, trustworthy machine learning, cybersecurity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security. https://app.cochat.ai/papers/10.1109/ijcnn48605.2020.9206780
- The paper
- Kuppa, A., & Le-Khac, N.-A. (2020). Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security. 2020 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn48605.2020.9206780
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.