Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
Open accessAbstract
To make Explainable AI (XAI) systems trustworthy, understanding harmful effects is just as important as producing well-designed explanations. In this paper, we address an important yet unarticulated type of negative effect in XAI. We introduce explainability pitfalls(EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We articulate the concept of EPs by demarcating it from dark patterns and highlighting the challenges arising from uncertainties around pitfalls. We situate and operationalize the concept using a case study that showcases how, despite best intentions, unsuspecting negative effects such as unwarranted trust in numerical explanations can emerge. We propose proactive and preventative strategies to address EPs at three interconnected levels: research, design, and organizational.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing XAI methods often do not match end-users' mental models, making explanations ineffective and sometimes harmful.
“Such explanations often do not match the end-users' mental models [56], making the explanations ineective [2,40,65] and at times even harmful [13,28,30,47,51].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2011) · Cited by 14,317 · see citing works
The Elaboration Likelihood Model of Persuasion
Petty, R. E., & Cacioppo, J. T. (1986) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60214-2 · Cited by 8,822 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
Muller, M. J., & Kuhn, S. (1993) · Communications of the ACM · doi:10.1145/153571.255960 · Cited by 909 · see citing works
The Dark (Patterns) Side of UX Design
Gray, C. M., Kou, Y., Battles, B., Hoggatt, J., & Toombs, A. L. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174108 · Cited by 871 · see citing works
Speculative design: crafting the speculation
Auger, J. (2013) · Digital Creativity · doi:10.1080/14626268.2013.767276 · Cited by 760 · see citing works
Langer, E. J., Blank, A., & Chanowitz, B. (1978) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.36.6.635 · Cited by 694 · see citing works
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 611 · see citing works
Wason, P. C., & Evans, J. S. B. T. (1974) · Cognition · doi:10.1016/0010-0277(74)90017-1 · Cited by 575 · see citing works
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning
Byrne, R. M. J. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/876 · Cited by 285 · see citing works
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274 · see citing works
Self-citationInterpretable Deep Models for ICU Outcome Prediction.
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2016) · PubMed Central · Cited by 240 · see citing works
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210 · see citing works
Self-citationVertesi, J. (2014) · Science Technology & Human Values · doi:10.1177/0162243913516012 · Cited by 185 · see citing works
The age of secrecy and unfairness in recidivism prediction
Rudin, C., Wang, C., & Coker, B. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.6ed64b30 · Cited by 177 · see citing works
The Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152 · see citing works
Material Signals: A Historical Sociology of High-Frequency Trading
MacKenzie, D. A. (2018) · American Journal of Sociology · doi:10.1086/697318 · Cited by 149 · see citing works
Dark Patterns in the Design of Games
Zagal, J. P., Björk, S., & Lewis, C. (2013) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 129 · see citing works
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112 · see citing works
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106 · see citing works
"Beautiful Seams": Strategic Revelations and Concealments
Inman, S., & Ribes, D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300508 · Cited by 81 · see citing works
Machine Learning Model Interpretability for Precision MedicinePreprint
Katuwal, G. J., & Chen, R. F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09045 · Cited by 74 · see citing works
Seamful Design for Location-Based Mobile Games
Broll, G., & Benford, S. (2005) · Lecture notes in computer science · doi:10.1007/11558651_16 · Cited by 59 · see citing works
Dark Patterns of Explainability, Transparency, and User Control for Intelligent Systems.
Chromik, M., Eiband, M., Völkel, S. T., & Buschek, D. (2019) · IUI Workshops · Cited by 42 · see citing works
The Who in Explainable AI: How AI Background Shapes Perceptions of AI ExplanationsPreprint
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M. J., & Riedl, M. O. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Self-citationColusso, L. F., Bennett, C. L., Gabriel, P., & Rosner, D. K. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3323690 · Cited by 6 · see citing works
Dark Patterns, Electronic Medical Records, and the Opioid EpidemicPreprint
Daniel Capurro, N., & Velloso, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08870 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [9][9] John Murawski. Mortgage providers look to ai to process home loans faster. Wall Street Journal , March 2019. -
- [12][12] Karen Hao. Ai is sending people to jail - and getting it wrong. MIT Technology Review , January 2019. -
- [13][13] Simone Stumpf, Adrian Bussone, and Dympna O'sullivan. Explanations considered harmful? user interactions with machine learning systems. In ACM SIGCHI Workshop on HumanCentered Machine Learning , 2016. -
- [24][24] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3-4):279-292, 1992. -
- [30][30] Matthew Chalmers and Ian MacColl. Seamful and seamless design in ubiquitous computing. In Workshop at the crossroads: The interaction of HCI and systems issues in UbiComp , volume 8, 2003. -
- [38][38] Phoebe Sengers, Kirsten Boehner, Shay David, and Joseph'Jofish' Kaye. Reflective design. In Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility , pages 49-58, 2005. -
Cited by
Sorted by citations
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · see citing works
Self-citationSchoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67 · see citing works
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69 · see citing works
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55 · see citing works
Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42 · see citing works
- Authors
- Ehsan, U., & Riedl, M. O.
- Affiliations
- Georgia Institute of Technology (US)
- Corresponding author
- Ehsan, Upol
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 5
- Issue
- 6
- Pages
- 100971-100971
- ISSN
- 2666-3899 (ISSN-L)
- arXiv
- 2109.12480
- PMC
- PMC11240172
- OpenAlex
- W4399670658
- Cited by
- 99
- References
- 32
- Citations by year
- 2026: 32 · 2025: 27 · 2024: 15 · 2023: 15 · 2022: 10
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Patterns (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1928586)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- explainable AI, dark patterns, AI explanations, trustworthy AI, artificial intelligence adoption, stakeholder empowerment, AI design, organizational responsibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
