CoChat
Sign Up
Cited by 37Explainable Artificial Intelligence (XAI)Human-Automation Interaction and SafetyEthics and Social Impacts of AI

Evaluating the impact of human explanation strategies on human-AI visual decision-making

Open access
Morrison, Katelyn, Shin, Donghoon, Holstein, Kenneth, Perer, Adam·2023·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3579481
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence (AI) is increasingly being deployed in high-stakes domains, such as disaster relief and radiology, to aid practitioners during the decision-making process. Explainable AI techniques have been developed and deployed to provide users insights into why the AI made certain predictions. However, recent research suggests that these techniques may confuse or mislead users. We conducted a series of two studies to uncover strategies that humans use to explain decisions and then understand how those explanation strategies impact visual decision-making. In our first study, we elicit explanations from humans when assessing and localizing damaged buildings after natural disasters from satellite imagery and identify four core explanation strategies that humans employed. We then follow up by studying the impact of these explanation strategies by framing the explanations from Study 1 as if they were generated by AI and showing them to a different set of decision-makers performing the same task. We provide initial insights on how causal explanation strategies improve humans' accuracy and calibrate humans' reliance on AI when the AI is incorrect. However, we also find that causal explanation strategies may lead to incorrect rationalizations when AI presents a correct assessment with incorrect localization. We explore the implications of our findings for the design of human-centered explainable AI and address directions for future work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3579481

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent work shows an evolution toward more rigorous, multifaceted trust assessments, in contrast to many earlier XAI studies that demonstrated technical feasibility and interpretability but did not empirically evaluate user trust or integration into real clinical workflows.

    “The evolution toward more rigorous, multifaceted trust assessments in recent work [53,63-65] stands in contrast to many earlier XAI studies [11,45,46,66,67], which often demonstrated technical feasibility and interpretability but did not empirically evaluate user trust or integration into real clinical workflows.”

    From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [65]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 53

  1. Amina Adadi and Mohammed Berrada. 2018. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access 6 (2018), 52138-52160. https://doi.org/10.1109/ACCESS.2018.2870052 · DOI · Paper page
  2. Josh Andres, Christine T. Wolf, Sergio Cabrero Barros, Erick Oduor, Rahul Nair, Alexander Kjærum, Anders Bech Tharsgaard, and Bo Schwartz Madsen. 2020. Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1-8. https://doi.org/10.1145/ 3334480.3382903 · DOI
  3. Joseph L Austerweil and Thomas L Griffiths. 2011. Seeking confirmation is rational for deterministic hypotheses. Cognitive Science 35, 3 (2011), 499-526. https://doi.org/10.1111/j.1551-6709.2010.01161.x · DOI
  4. Andrea Brennen. 2020. What Do People Really Want When They Say They Want "Explainable AI?" We Asked 60 Stakeholders. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20). Association for Computing Machinery, 1-7. https://doi.org/10.1145/3334480.3383047 · DOI
  5. Ángel Alexander Cabrera, Abraham J Druck, Jason I Hong, and Adam Perer. 2021. Discovering and Validating AI Errors With Crowdsourced Failure Reports. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-22. https://doi.org/10.1145/3479569 · DOI
  6. Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019. "Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction 3, CSCW (Nov 2019), 1-24. https://doi.org/10.1145/3359206 · DOI · Paper page
  7. David S Channin, Pattanasak Mongkolwat, Vladimir Kleper, and Daniel L Rubin. 2009. The annotation and image mark-up project. https://doi.org/10.1148/radiol.2533090135 · DOI
  8. Eric Chu, Deb Roy, and Jacob Andreas. 2020. Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction. (Jul 2020). https://arxiv.org/abs/2007.12248v1 arXiv: 2007.12248v1. · arXiv
  9. Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. 2018. Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES '18). ACM, 81-87. https://doi.org/10.1145/3278721.3278736 · DOI
  10. Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.). Springer International Publishing, 449-466. https://doi.org/10.1007/978-3-030-60117-1_33 · DOI · Paper page
  11. Upol Ehsan, Philipp Wintersberger, Q. Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O. Riedl. 2021. Operationalizing Human-Centered Perspectives in Explainable AI. Association for Computing Machinery, 1-6. https://doi.org/10.1145/3411763.3441342 · DOI
  12. Ziba Gandomkar and Claudia Mello-Thoms. 2019. Visual search in breast imaging. The British journal of radiology 92, 1102 (2019), 20190057. https://doi.org/10.1259/bjr.20190057 · DOI
  13. Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations. arXiv:1904.07451 [cs, stat] (Jun 2019). http://arxiv.org/abs/1904.07451 arXiv: 1904.07451. · arXiv
  14. Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. 2019. Creating xBD: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 10-17. https://doi.org/10.1184/R1/8135576.v1 · DOI
  15. Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Gener- ating Visual Explanations. In Computer Vision -ECCV 2016 (Lecture Notes in Computer Science), Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, 3-19. https://doi.org/10.1007/978-3-319- 46493-0_1 · DOI · Paper page
  16. Ralph Hertwig and Gerd Gigerenzer. 1999. The 'conjunction fallacy' revisited: How intelligent inferences look like reasoning errors. Journal of behavioral decision making 12, 4 (1999), 275-305. https://doi.org/10.1002/(sici)1099- 0771(199912)12:4%3C275::aid-bdm323%3E3.0.co;2-m · DOI
  17. Kenneth Holstein and Vincent Aleven. 2022. Designing for human-AI complementarity in K-12 education. AI Magazine 43, 2 (jun 2022), 239-248. https://doi.org/10.1002/aaai.12058 · DOI · Paper page
  18. Neal Jean, Marshall Burke, Michael Xie, W. Matthew Davis, David B. Lobell, and Stefano Ermon. 2016. Combining satellite imagery and machine learning to predict poverty. Science 353, 6301 (Aug 2016), 790-794. https://doi.org/10. 1126/science.aaf7894 · DOI · Paper page
  19. Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. 2019. XRAI: Better Attributions Through Regions. https://arxiv.org/abs/1906.02825 arXiv: 1906.02825. · arXiv
  20. Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-14. https://doi.org/10.1145/3313831. 3376219 · DOI · Paper page
  21. Asim B. Khajwal and Arash Noshadravan. 2021. An uncertainty-aware framework for reliable disaster damage assessment via crowdsourcing. International Journal of Disaster Risk Reduction 55 (Mar 2021), 102110. https://doi.org/ 10.1016/j.ijdrr.2021.102110 · DOI
  22. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237-293. https://doi.org/10.3386/w23180 · DOI · Paper page
  23. Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471 (2021). https://arxiv.org/abs/2112.11471 · arXiv
  24. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences 40 (2017). https://doi.org/10.1017/S0140525X16001837 · DOI · Paper page
  25. Himabindu Lakkaraju and Osbert Bastani. 2020. "How do I fool you?" Manipulating User Trust via Misleading Black Box Explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 79-85. https://doi.org/10. 1145/3375627.3375833 · DOI · Paper page
  26. Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Thirty-first aaai conference on artificial intelligence. https://doi.org/10.1609/aaai.v31i1.10821 · DOI
  27. Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T. Freeman, Phillip Isola, Amir Globerson, Michal Irani, and Inbar Mosseri. 2021. Explaining in Style: Training a GAN to explain a classifier in StyleSpace. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE. https://doi.org/10.1109/ iccv48922.2021.00073 · DOI
  28. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50-80. https://doi.org/10.1518/hfes.46.1.50.30392 · DOI · Paper page
  29. Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, 1-15. https://doi.org/10.1145/3313831.3376590 · DOI · Paper page
  30. Q. Vera Liao and Kush R. Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. (Dec 2021). http://arxiv.org/abs/2110.10790 arXiv: 2110.10790. · arXiv
  31. Zhong Qiu Lin, Mohammad Javad Shafiee, Stanislav Bochkarev, Michael St Jules, Xiao Yu Wang, and Alexander Wong. 2019. Do explanations reflect decisions? A machine-centric strategy to quantify the performance of explainability algorithms. arXiv preprint arXiv:1910.07387 (2019). https://arxiv.org/abs/1910.07387 · arXiv
  32. Zachary C Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31-57. https://doi.org/10.1145/3236386.3241340 · DOI · Paper page
  33. Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17).
  34. Curran Associates Inc., Red Hook, NY, USA, 4768-4777. https://dl.acm.org/doi/pdf/10.5555/3295222.3295230 · DOI
  35. Sahar S. Matin and Biswajeet Pradhan. 2021. Earthquake-Induced Building-Damage Mapping Using Explainable AI (XAI). Sensors 21, 13 (2021). https://doi.org/10.3390/s21134489 · DOI
  36. Mahsan Nourani, Samia Kabir, Sina Mohseni, and Eric D. Ragan. 2019. The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems. 7 (Oct 2019), 97-105. https://ojs.aaai. org/index.php/HCOMP/article/view/5284
  37. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature Visualization. Distill 2, 11 (Nov 2017), e7. https://doi.org/10.23915/distill.00007 · DOI · Paper page
  38. Barak Oshri, Annie Hu, Peter Adelson, Xiao Chen, Pascaline Dupas, Jeremy Weinstein, Marshall Burke, David Lobell, and Stefano Ermon. 2018. Infrastructure Quality Assessment in Africa using Satellite Imagery and Deep Learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Jul 2018), 616-625. https://doi.org/10.1145/3219819.3219924 · DOI
  39. Bhavik N Patel, Louis Rosenberg, Gregg Willcox, David Baltaxe, Mimi Lyons, Jeremy Irvin, Pranav Rajpurkar, Timothy Amrhein, Rajan Gupta, Safwan Halabi, et al. 2019. Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine 2, 1 (2019), 1-10. https://doi.org/10.1038/s41746-019-0189-7 · DOI · Paper page
  40. Mauricio Reyes, Raphael Meier, Sérgio Pereira, Carlos A Silva, Fried-Michael Dahlweid, Hendrik von Tengg-Kobligk, Ronald M Summers, and Roland Wiest. 2020. On the interpretability of artificial intelligence in radiology: challenges and opportunities. Radiology: Artificial Intelligence 2, 3 (2020), e190043. https://doi.org/10.1148/ryai.2020190043 · DOI · Paper page
  41. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). 1135-1144. https://doi.org/10.1145/2939672.2939778 · DOI · Paper page
  42. Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G. Blankenberg, Andrew Y. Ng, Matthew P. Lungren, and Pranav Rajpurkar. 2021. Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation. medRxiv (Mar 2021). https://doi.org/10.1101/2021.02.28.21252634v1 · DOI
  43. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. International Journal of Computer Vision 128, 2 (Feb 2020), 336-359. https://doi.org/10.1007/s11263-019-01228-7 · DOI · Paper page
  44. Hidehiko Shishido, Koyo Kobayashi, Yoshinari Kameda, and Itaru Kitahara. 2021. Method to Generate Building Damage Maps by Combining Aerial Image Processing and Crowdsourcing. Journal of Disaster Research 16, 5 (Aug 2021), 827-839. https://doi.org/10.20965/jdr.2021.p0827 · DOI
  45. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. (Apr 2014). http://arxiv.org/abs/1312.6034 arXiv: 1312.6034. · arXiv
  46. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 180-186. https://doi.org/10.1145/3375627.3375830 · DOI · Paper page
  47. Richard Szeliski. 2010. Computer vision: algorithms and applications. Springer Science & Business Media. https: //doi.org/10.1007/978-3-030-34372-9 · DOI
  48. Jerome Theau. 2011. Change detection. In Springer Handbook of Geographic Information. Springer, 75-94. https: //doi.org/10.1007/978-0-387-35973-1_129 · DOI
  49. Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. arXiv:1711.00399 [cs] (Mar 2018). http://arxiv.org/abs/1711.00399 arXiv: 1711.00399. · arXiv
  50. Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1-6. https://doi.org/10.1145/3334480. 3381069 · DOI · Paper page
  51. Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-15. https://doi.org/10.1145/ 3290605.3300831 · DOI · Paper page
  52. Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. 2020. Using publicly available satellite imagery and deep learning to understand economic well-being in Africa. Nature Communications 11, 1 (May 2020), 2583. https://doi.org/10.1038/s41467-020-16185-w · DOI · Paper page
  53. Alexandra Zytek, Dongyu Liu, Rhema Vaithianathan, and Kalyan Veeramachaneni. 2021. Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making. (Sep 2021). http: //arxiv.org/abs/2103.02071 arXiv: 2103.02071. · arXiv

Cited by

Sorted by citations

Authors
Morrison, K., Shin, D., Holstein, K., & Perer, A.
Affiliations
  • Carnegie Mellon University (US)
  • University of Washington, Seattle, WA, USA
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2023-04-14
Volume
7
Issue
CSCW1
Pages
1-37
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3579481
OpenAlex
W4365998993
Cited by
37
References
11
Citations by year
2026: 10 · 2025: 12 · 2024: 15
FWCI
3.27
Type
journal article
Language
en
Open access
Licence
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by-nd)
Funders
Center for Machine Learning and Health - Internal Funding
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
explainable AI, human-AI decision-making, visual decision making, causal explanation, natural disaster damage assessment, satellite imagery, human-centered AI design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review