"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Open accessAbstract
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a human interpretable manner. There has been recent concern that a high-fidelity explanation of a black box ML model may not accurately reflect the biases in the black box. As a consequence, explanations have the potential to mislead human users into trusting a problematic black box. In this work, we rigorously explore the notion of misleading explanations and how they influence user trust in black box models. Specifically, we propose a novel theoretical framework for understanding and generating misleading explanations, and carry out a user study with domain experts to demonstrate how these explanations can be used to mislead users. Our work is the first to empirically establish how user trust in black box models can be manipulated via misleading explanations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The untrustworthy coach did not need to generate misleading explanations.
“It did not even need to generate misleading explanations [30,46].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [30]
Existing XAI methods often do not match end-users' mental models, making explanations ineffective and sometimes harmful.
“Such explanations often do not match the end-users' mental models [56], making the explanations ineective [2,40,65] and at times even harmful [13,28,30,47,51].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [30]
It is unclear whether untrustworthy AI can deceive end-users and gain their trust by means other than deceptive explanations.
“For example, it is unclear if there is a way for untrustworthy AI to deceive end-users and gain their trust other than by using deceptive explanations [30,46].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [30]
Untrustworthy AI could generate explanations itself in order to deceive willfully, or use existing methods to inhibit reliance on trustworthy AI.
“untrustworthy AI could generate similar explanations itself to willfully deceive [30,46] or even use existing methods to inhibit reliance on AI [9] against trustworthy AI.”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Fast algorithms for mining association rules
Agrawal, R., & Srikant, R. (1998) · Cited by 10,706 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1× · see citing works
“, ensuring that the explanations accurately mimic the predictions of black box model [12, 17, 18].”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
Self-citationThe budgeted maximum coverage problem
Khuller, S., Moss, A., & Naor, J. S. (1999) · Information Processing Letters · doi:10.1016/s0020-0190(99)00031-9 · Cited by 1,009 · see citing works
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · see citing works
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · see citing works
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · see citing works
Self-citationHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545 · see citing works
Self-citationFaithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 261 · see citing works
Self-citationNon-monotone submodular maximization under matroid and knapsack constraints
Lee, J., Mirrokni, V., Nagarajan, V., & Sviridenko, M. I. (2009) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1536414.1536459 · Cited by 234 · see citing works
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · see citing works
Non-monotone submodular maximization under matroid and knapsack constraintsPreprint
Lee, J., Mirrokni, V., Nagarjan, V., et al. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0902.0353 · Cited by 115 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Osbert Bastani, Carolyn Kim, and Hamsa Bastani. 2017. Interpretability via model extraction. arXiv preprint arXiv:1706.09773 (2017). · arXiv
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of inter- pretable machine learning. arXiv preprint arXiv:1702.08608 (2017). · arXiv
- Carolyn Kim and Osbert Bastani. 2019. Learning Interpretable Models with Causal Guarantees. arXiv preprint arXiv:1901.08576 (2019). · arXiv
- Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006 (2019). · arXiv
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
- Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GPDR. Harv. JL & Tech. 31 (2017), 841.
- Qingyuan Zhao and Trevor Hastie. 2019. Causal interpretations of black-box models. Journal of Business & Economic Statistics just-accepted (2019), 1-19.
Cited by
Sorted by citations
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · see citing works
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · see citing works
Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161 · see citing works
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129 · see citing works
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127 · see citing works
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117 · see citing works
Danry, V., Pataranutaporn, P., Mao, Y., & Maes, P. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580672 · Cited by 110 · see citing works
Does Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106 · see citing works
Analyzing and assessing explainable AI models for smart agriculture environments
Cartolano, A., Cuzzocrea, A., & Pilato, G. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17978-z · Cited by 92 · see citing works
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69 · see citing works
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55 · see citing works
Integrity-based Explanations for Fostering Appropriate Trust in AI Agents
Mehrotra, S., Jorge, C. C., Jonker, C. M., & Tielman, M. L. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3610578 · Cited by 51 · see citing works
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37 · see citing works
Self-fulfilling prophecies and machine learning in resuscitation science
De-Arteaga, M., & Elmer, J. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2022.10.014 · Cited by 28 · see citing works
Don’t Be Fooled: The Misinformation Effect of Explanations in Human–AI Collaboration
Spitzer, P., Holstein, J., Morrison, K., Holstein, K., Satzger, G., & Kühl, N. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2574511 · Cited by 12 · see citing works
Interpretable AI/ML for High-stakes Tasks with Human-in-the-loop: Critical Review and Future Trends
Kovalerchuk, B. (2024) · doi:10.21203/rs.3.rs-3989807/v1 · Cited by 9 · see citing works
Study on the Helpfulness of Explainable Artificial Intelligence
Labarta, T., Kulicheva, E., Froelian, R., Geißler, C., Melman, X., & von Klitzing, J. (2024) · Communications in Computer and Information Science · doi:10.1007/978-3-031-63803-9_16 · Cited by 8 · see citing works
- Authors
- Lakkaraju, H., & Bastani, O.
- Affiliations
- Harvard University (US)
- University of Pennsylvania (US)
- Venue
- Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2020-02-07
- Pages
- 79-85
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W3005073185
- Cited by
- 241
- References
- 16
- Citations by year
- 2026: 19 · 2025: 37 · 2024: 52 · 2023: 36 · 2022: 39 · 2021: 34 · 2020: 19 · 2019: 4 · 2018: 1
- FWCI
- 12.05
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- Funders
- Google (Research Award)
- National Science Foundation (NSF) (CCF-1910769)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, user trust, algorithmic bias, trust manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
