CoChat
Sign Up
Cited by 123Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIInnovative Human-Technology Interaction

The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

Open access
Ehsan, Upol, Passi, Samir, Liao, Q. Vera, Chan, Larry, Lee, I-Hsiang, Muller, Michael, Riedl, Mark O·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613904.3642474
Open this paper in CoChat — free accountOpen PDF

Abstract

Explainability of AI systems is critical for users to take informed actions. Understanding who opens the black-box of AI is just as important as opening it. We conduct a mixed-methods study of how two different groups—people with and without AI background—perceive different types of AI explanations. Quantitatively, we share user perceptions along five dimensions. Qualitatively, we describe how AI background can influence interpretations, elucidating the differences through lenses of appropriation and cognitive heuristics. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design. Carrying critical implications for the field of XAI, our findings showcase how AI generated explanations can have negative consequences despite best intentions and how that could lead to harmful manipulation of trust. We propose design interventions to mitigate them.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642474

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 101 · Cited by 123

Research works printed but not linked: 55

  1. [5][5] Alan Agresti. 2003. Categorical data analysis . Vol. 482. John Wiley & Sons. -
  2. [12][12] David Bawden et al. 2008. Origins and concepts of digital literacy. Digital literacies: Concepts, policies and practices 30, 2008 (2008), 17-32. -
  3. [13][13] Tara S Behrend, David J Sharek, Adam W Meade, and Eric N Wiebe. 2011. The viability of crowdsourcing for survey research. Behavior research methods 43, 3 (2011), 800. -
  4. [24][24] Kathy Charmaz. 2014. Constructing Grounded Theory (Introducing Qualitative Methods series) 2nd Edition . Sage, London. -
  5. [25][25] Zhengping Che, Sanjay Purushotham, Robinder Khemani, and Yan Liu. 2016. Interpretable deep models for ICU outcome prediction. In AMIA Annual Symposium Proceedings , Vol. 2016. American Medical Informatics Association, 371. -
  6. [26][26] Jim Q Chen and Sang M Lee. 2003. An exploratory cognitive DSS for strategic decision making. Decision support systems 36, 2 (2003), 147-160. -
  7. [30][30] Michael Chromik, Malin Eiband, Sarah Theres Völkel, and Daniel Buschek. 2019. Dark Patterns of Explainability, Transparency, and User Control for Intelligent Systems.. In IUI workshops , Vol. 2327. -
  8. [31][31] Mark Core, H. Chad Lane, Michael van Lent, Dave Gomboc, Steve Solomon, and Milton Rosenberg. 2006. Building Explainable Artificial Intelligence Systems. In Proceedings of the 18th Innovative Applications of Artificial Intelligence Conference . Boston, MA. -
  9. [35][35] Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. 2018. Boolean Decision Rules via Column Generation. Advances in Neural Information Processing Systems 31 (2018), 4655-4665. -
  10. [36][36] Fred D Davis, Richard P Bagozzi, and Paul R Warshaw. 1989. User acceptance of computer technology: a comparison of two theoretical models. Management science 35, 8 (1989), 982-1003. -
  11. [39][39] Andrea A DiSessa. 2001. Changing minds: Computers, learning, and literacy . Mit Press. -
  12. [40][40] Alan Dix. 2007. Designing for appropriation. In Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK 21 . 1-4. -
  13. [43][43] Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. 2018. Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, Ethics, and Society .
  14. [44][44] Upol Ehsan, Q Vera Liao, Samir Passi, Mark O Riedl, and Hal Daume III. 2022. Seamful XAI: Operationalizing Seamful Design in Explainable AI. arXiv preprint arXiv:2211.06753 (2022). - · arXiv
  15. [45][45] Upol Ehsan and Mark O Riedl. 2020. Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach. arXiv preprint arXiv:2002.01092 (2020). - · arXiv
  16. [55][55] Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable Active Learning (XAL) Toward AI Explanations as Interfaces for Machine Teachers. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1-28. -
  17. [56][56] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An approach to evaluating interpretability of machine learning. arXiv preprint arXiv:1806.00069 (2018). - · arXiv
  18. [57][57] Barney Glaser and Anselm Strauss. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research . Aldine Transactions, Chicago. -
  19. [61][61] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1-42. -
  20. [63][63] David J Hauser and Norbert Schwarz. 2016. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior research methods 48, 1 (2016), 400-407. -
  21. [65][65] Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the 2019 CHI conference on human factors in computing systems . 1-13. -
  22. [66][66] Myles Hollander, Douglas A Wolfe, and Eric Chicken. 2013. Nonparametric statistical methods . Vol. 751. John Wiley & Sons. -
  23. [70][70] James Jaccard. 2001. Interaction effects in logistic regression . Vol. 135. SAGE Publications, Incorporated. -
  24. [74][74] WLewis Johnson. 1994. Agents that Learn to Explain Themselves.. In AAAI . 1257-1263. -
  25. [78][78] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-14. -
  26. [83][83] Alina Krischkowsky. 2018. Technology in Uses: Negotiating the in-between of designer | user and design | use . Ph.D. Dissertation. University of Salzburg, Salzburg, Austria. -
  27. [86][86] Kathryn Ann Lambe, Gary O'Reilly, Brendan D Kelly, and Sarah Curristan. 2016. Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review. BMJ quality & safety 25, 10 (2016), 808-820. -
  28. [87][87] Ellen J Langer, Arthur Blank, and Benzion Chanowitz. 1978. The mindlessness of ostensibly thoughtful action: The role of" placebic" information in interpersonal interaction. Journal of personality and social psychology 36, 6 (1978), 635. -
  29. [89][89] Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on . IEEE, 203-210. -
  30. [92][92] Q Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael Muller, N Sadat Shami, and Werner Geyer. 2018. All work and no play? Conversations with a Question-and-Answer Chatbot in the Wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . 1-13. -
  31. [95][95] Peter Lipton. 2001. What good is an explanation? In Explanation . Springer, 43-59. -
  32. [97][97] Leib Litman, Jonathan Robinson, and Tzvi Abberbock. 2017. TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods 49, 2 (2017), 433-442. -
  33. [100][100] Tania Lombrozo. 2012. Explanation and abductive inference. Oxford handbook of thinking and reasoning (2012), 260-276. -
  34. [105][105] Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems . 4765-4774. -
  35. [108][108] Ioana E Marinescu, Patrick N Lawlor, and Konrad P Kording. 2018. Quasi-experimental causality in neuroscience and behavioural research. Nature human behaviour 2, 12 (2018), 891-898. -
  36. [114][114] Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2020. A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. arXiv:1811.11839 [cs.HC] - · arXiv
  37. [120][120] Clifford Nass, BJ Fogg, and Youngme Moon. 1996. Can computers be teammates? International Journal of Human-Computer Studies 45, 6 (1996), 669-678. -
  38. [123][123] Don Norman. 2013. The design of everyday things: Revised and expanded edition . Basic books. -
  39. [124][124] Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In 26th International Conference on Intelligent User Interfaces . 340-350. -
  40. [128][128] Samir Passi and Steven J. Jackson. 2017. Data Vision: Learning to See Through Algorithmic Abstraction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17) . ACM Press, 2436-2447. -
  41. [131][131] Samir Passi and Mihaela Vorvoreanu. 2022. Overreliance on AI: Literature Review . Technical Report MSR-TR-2022-12. Microsoft. https: //www.microsoft.com/en-us/research/publication/overreliance-on-ai-literature-review/ -
  42. [138][138] Antti Salovaara. 2008. Inventing new uses for tools: a cognitive foundation for studies on appropriation. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments (2008). -
  43. [142][142] Edward Shortliffe. 2012. Computer-based medical consultations: MYCIN . Vol. 2. Elsevier. -
  44. [145][145] Jon Sprouse. 2011. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior research methods 43, 1 (2011), 155-167. -
  45. [148][148] Ernest T Stringer. 2007. Action research third edition. (2007). -
  46. [149][149] Harini Suresh and John V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 (2019). - · arXiv
  47. [150][150] William R Swartout. 1983. XPLAIN: A system for creating and explaining expert consulting programs. Artificial intelligence 21, 3 (1983), 285-325. -
  48. [152][152] Linda Tickle-Degnen and Robert Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry 1, 4 (1990), 285-293. -
  49. [156][156] Michael van Lent, , Paul Carpenter, Ryan McAlinden, and Paul Brobst. 2005. Increasing Replayability with Deliberative and Reactive Planning. In 1st Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE) . Maria Del Rey, California, 135-140. -
  50. [157][157] Michael Van Lent, William Fisher, and Michael Mancuso. 2004. An explainable artificial intelligence system for small-unit tactical behavior. In Proceedings of the national conference on artificial intelligence . Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 900-907. -
  51. [160][160] Viswanath Venkatesh, Michael G Morris, Gordon B Davis, and Fred D Davis. 2003. User acceptance of information technology: Toward a unified view. MIS quarterly (2003), 425-478. -
  52. [163][163] Peter C Wason and J St BT Evans. 1974. Dual processes in reasoning? Cognition 3, 2 (1974), 141-154. -
  53. [164][164] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3-4 (1992), 279-292. -
  54. [165][165] Dennis Wei, Sanjeeb Dash, Tian Gao, and Oktay Gunluk. 2019. Generalized linear rule models. In International Conference on Machine Learning . PMLR, 6687-6696. -
  55. [168][168] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning . 2048-2057. -

Authors
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O.
Affiliations
  • Georgia Institute of Technology (US)
  • Microsoft (United States) (US)
  • Microsoft (Canada) (CA)
  • Illumio, United States
  • IBM (United States) (US)
ORCID
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-32
DOI
10.1145/3613904.3642474
OpenAlex
W4396833700
Cited by
123
References
101
Citations by year
2026: 47 · 2025: 44 · 2024: 8 · 2023: 16 · 2022: 7 · 2021: 1
FWCI
29.22
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, public-domain)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1928586)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
explainable AI, AI explanations, trust in AI, cognitive heuristics, appropriations, AI-generated explanations, trust manipulation, explanation design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review