Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Open accessAbstract
As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is a growing body of statistical work contesting the use of SHAP values for feature explanations.
“Lastly, there is a growing body of statistical work contesting the use of SHAP values for feature explanations [24][25][26] .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [26]
Post-hoc methods could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled.
“There are also concerns that post-hoc methods could expose the original models to adversarial attacks [18] or could lead to the generation of classifiers whose post-hoc explanations could be arbitrarily controlled [63].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [63]
There is some concern around the reliability of post-hoc explanations.
“There is some concern around the reliability of post-hoc explanations [63,90].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [63]
Prior studies show that input influence measures like SHAP can be fooled into stating that a protected attribute has no influence on a model.
“Prior studies show that input influence measures like SHAP can be fooled into stating that a protected attribute has no influence on a model [49].”
From Marrying Fairness and Explainability in Supervised Learning (2022) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
Self-citationThe mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1× · see citing works
“For instance, anchors [21] use (non-linear) rules to express more actionable local explanations.”
BackgroundSelf-citationKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Neural Information Processing Systems (NIPS)
Shang, X., Zhu, Z., Leimkuhler, B., & Storkey, A. (2015) · Cited by 831 · see citing works
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · see citing works
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · see citing works
The Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392 · see citing works
A data-driven software tool for enabling cooperative information sharing among police departments
Redmond, M., & Baveja, A. (2002) · European Journal of Operational Research · doi:10.1016/s0377-2217(01)00264-8 · Cited by 211 · see citing works
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · see citing works
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278725 · Cited by 127 · see citing works
Fooling Neural Network Interpretations via Adversarial Model ManipulationPreprint
Heo, J., Joo, S., & Moon, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02041 · Cited by 126 · see citing works
Global Explanations of Neural Networks
Ibrahim, M., Louie, M., Modarres, C., & Paisley, J. W. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314230 · Cited by 126 · see citing works
Fairwashing: the risk of rationalization
Aïvodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., & Tapp, A. (2019) · arXiv · doi:10.48550/arxiv.1901.09749 · Cited by 54 · see citing works
Mapping chemical performance on molecular structures using locally interpretable explanationsPreprint
Whitmore, L. S., George, A., & Hudson, C. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07443 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [7]7. Radwa Elshawi, Mouaz H Al-Mallah, and Sherif Sakr. 2019. On the interpretability of machine learning-based model for predicting hypertension. BMC medical informatics and decision making 19, 1 (2019), 146.
- [12]12. Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. U. Pa. L. Rev. 165 (2016), 633.
- [16]16. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765-4774.
- [17]17. Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining expla- nations in AI. In Proceedings of the conference on fairness, accountability, and transparency. ACM, 279-288.
- [20]20. General Data Protection Regulation. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46. Official Journal of the European Union (OJ) 59, 1-88 (2016), 294.
Cited by
Sorted by citations
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · see citing works
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · see citing works
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
Zhang, Z., Hamadi, H. A., Damiani, E., Yeun, C. Y., & Taher, F. (2022) · IEEE Access · doi:10.1109/access.2022.3204051 · Cited by 450 · see citing works
Moustafa, N., Koroniotis, N., Keshk, M., Zomaya, A. Y., & Tari, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3280465 · Cited by 266 · see citing works
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
Hooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264 · see citing works
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · see citing works
“Slack et al. (2020) demonstrate how to exploit this sensitivity by devising models which illegally discriminate on some protected feature for in-distribution samples, but exhibit different behavior on the out-of-distribution samples used by KernelSHAP so as to simulate “fairness” in the resulting explanations.”
BackgroundAbbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236 · see citing works
Explainable AI for 6G Use Cases: Technical Aspects and Research Challenges
Wang, S., Qureshi, M. A., Miralles-Pechuán, L., Huynh-The, T., Gadekallu, T. R., & Liyanage, M. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3386872 · Cited by 170 · see citing works
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167 · see citing works
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144 · see citing works
A Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139 · see citing works
“In particular, post-hoc XAI techniques such as LIME and SHAP are easily manipulated [199].”
MethodDeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
Han, D., Wang, Z., Chen, W., Zhong, Y., Wang, S., Zhang, H., Yang, J., Shi, X., & Yin, X. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484589 · Cited by 121 · see citing works
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
Bell, A., Solano-Kamaiko, I., Nov, O., & Stoyanovich, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533090 · Cited by 103 · see citing works
Analyzing and assessing explainable AI models for smart agriculture environments
Cartolano, A., Cuzzocrea, A., & Pilato, G. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17978-z · Cited by 92 · see citing works
How Interpretable and Trustworthy are GAMs?
Chang, C.-H., Tan, S., Lengerich, B., Goldenberg, A., & Caruana, R. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467453 · Cited by 66 · see citing works
Principles and Practice of Explainable Machine Learning
Belle, V., & Papantonis, I. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.688969 · Cited by 99 · see citing works
Bell, A., Nov, O., & Stoyanovich, J. (2023) · Data & Policy · doi:10.1017/dap.2023.8 · Cited by 44 · see citing works
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41 · see citing works
Explainable artificial intelligence, lawyer's perspective
Górski, Ł., & Ramakrishna, S. (2021) · Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law · doi:10.1145/3462757.3466145 · Cited by 40 · see citing works
Marrying Fairness and Explainability in Supervised Learning
Grabowicz, P. A., Perello, N., & Mishra, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533236 · Cited by 37 · see citing works
Is Genre Enough? A Theory of Genre Signaling as Generative AI Rhetoric
Omizo, R., & Hart-Davidson, B. (2024) · Rhetoric Society Quarterly · doi:10.1080/02773945.2024.2343615 · Cited by 16 · see citing works
Nomological Deductive Reasoning for Trustworthy, Human-Readable, and Actionable AI Outputs
Hakizimana, G., & Ledezma Espino, A. (2025) · Algorithms · doi:10.3390/a18060306 · Cited by 2 · see citing works
- Authors
- Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H.
- Affiliations
- University of California, Irvine (US)
- Harvard University (US)
- Venue
- Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2020-02-07
- Pages
- 180-186
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W3005086430
- Semantic Scholar
- 653864b10564ab4712c07a3d4043a1d794b13c46
- Cited by
- 900
- References
- 19
- Citations by year
- 2026: 207 · 2025: 180 · 2024: 162 · 2023: 134 · 2022: 85 · 2021: 103 · 2020: 27 · 2019: 2
- FWCI
- 33.32
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- Funders
- National Science Foundation (#IIS-1756023)
- Allen Institute of Artificial Intelligence (AI2)
- Google (Research Award)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- lime, SHAP, post-hoc explanation, explainable AI, algorithmic bias, scaffolding technique, input perturbation, COMPAS, racial bias, black-box machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
