CoChat
Sign Up
Cited by 81Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIHuman-Automation Interaction and Safety

In search of verifiability: Explanations rarely enable complementary performance in AI‐advised decision making

Open access
Fok, Raymond, Weld, Daniel S.·2024·AI Magazine
DOI 10.1002/aaai.12182
Open this paper in CoChat — free accountOpen PDF

Abstract

The current literature on AI‐advised decision making—involving explainable AI systems advising human decision makers—presents a series of inconclusive and confounding results. To synthesize these findings, we propose a simple theory that elucidates the frequent failure of AI explanations to engender appropriate reliance and complementary decision making performance. In contrast to other common desiderata, for example, interpretability or spelling out the AI's reasoning process, we argue that explanations are only useful to the extent that they allow a human decision maker to verify the correctness of the AI's prediction. Prior studies find in many decision making contexts that AI explanations do not facilitate such verification. Moreover, most tasks fundamentally do not allow easy verification, regardless of explanation method, limiting the potential benefit of any type of explanation. We also compare the objective of complementary performance with that of appropriate reliance, decomposing the latter into the notions of outcome‐graded and strategy‐graded reliance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/aaai.12182

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing reliance measures are restricted to structured, outcome-dependent decision-making tasks and do not account for complex, non-binary human-AI collaboration scenarios.

    “Thus, the two-dimensional reliance construct is restricted to structured, outcomedependent decision-making tasks and does not account for more complex, non-binary human-AI collaboration scenarios (Fok & Weld, 2024).”

    From Measuring undergraduate students’ reliance on generative AI during problem-solving: Scale development and validation. (2025) · cites this paper as Fok & Weld, 2024

    The Figure 6 discussion says outcome-graded reliance depends on post hoc correctness, supporting that part of the claim. But it also proposes strategy-graded reliance, which is not outcome-dependent, and applies the reliance definitions to an image-classification task with 1,000 possible classes. The paper therefore does not support describing both notions as restricted to outcome-dependent, binary tasks. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Reliance is appropriate when it is rooted in reasonable prior thought processes, even if outcomes are not optimal.

    “Based on strategy-­ graded reliance, reliance is appropriate when it is rooted in reasonable prior thought processes, even if the outcomes are not optimal.”

    From The role of critical thinking on undergraduates' reliance behaviours on generative <scp>AI</scp> in problem‐solving (2025) · cites this paper as Fok & Weld, 2024

  • Strategy-graded reliance emphasizes the reasoning process leading to reliance behaviours, focusing on thoughtful engagement with AI tools, and this cognitive process can be captured by critical thinking.

    “Strategy-­ graded reliance emphasizes the reasoning process leading to reliance behaviours, focusing on thoughtful engagement with AI tools, and this cognitive process can be captured by critical thinking.”

    From The role of critical thinking on undergraduates' reliance behaviours on generative <scp>AI</scp> in problem‐solving (2025) · cites this paper as Fok & Weld, 2024

  • The findings provide empirical support for the strategy-graded reliance framework, shifting the focus from outcome-based reliance measures to those grounded in reasoning processes.

    “In sum, these findings provide empirical support for the strategy-­ graded reliance framework, shifting the focus from outcome-­ based reliance measures to those grounded in reasoning processes (Fok &amp; Weld, 2024).”

    From The role of critical thinking on undergraduates' reliance behaviours on generative <scp>AI</scp> in problem‐solving (2025) · cites this paper as Fok & Weld, 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Alqaraawi, Ahmed, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020. "Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study." In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI '20), Cagliari, Italy, 275-85. New York, NY, USA: Association for Computing Machinery.
  2. Besmira Nushi Bansal, Gagan, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019a. "Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7(1): 2-11.
  3. Bell, Andrew, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich. 2022. "It's Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy." In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT'22), Seoul, Republic of Korea, 248-66. New York, NY, USA: Association for Computing Machinery.
  4. Bilgic, Mustafa, and Raymond J. Mooney. 2005. "Explaining Recom- mendations: Satisfaction vs. Promotion." In Proceedings of Beyond Personalization 2005: A Workshop on the Next Stage of Recom- mender Systems Research at the 2005 International Conference on Intelligent User Interfaces (IUI '05), 1-6. San Diego, CA: Association for Computing Machinery.
  5. Bowman, Samuel R., Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil ė Lukošiūt ė, et al. 2022. "Measuring Progress on Scalable Oversight for Large Language Models." arXiv preprint arXiv:2211.03540. · arXiv
  6. Buçinca, Zana, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. "Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems." In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI '20), Cagliari, Italy, 454-64. New York, NY, USA: Association for Computing Machinery.
  7. Bunt, Andrea, Matthew Lount, and Catherine Lauzon. 2012. "Are Explanations Always Important? A Study of Deployed, Low-Cost Intelligent Interactive Systems." In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (IUI'12), Lisbon, Portugal, 169-78. New York, NY, USA: Association for Computing Machinery.
  8. Carey, Harold R., and Patrick R. Laughlin. 2012. "Groups Perform Better than the Best Individuals on Letters-to-Numbers Prob- lems: Effects of Induced Strategies." Group Processes & Intergroup Relations 15(2): 231-42.
  9. Chandrasekaran, Arjun, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018. "Do Explanations Make VQA Models More Predictable to a Human?" In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, 1036-42. Brussels, Belgium: Association for Computational Linguistics.
  10. Chen, Chacha, Shi Feng, Amit Sharma, and Chenhao Tan. 2023. "Machine Explanations and Human Understanding." In Proceed- ings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT'23), Chicago, IL, USA, 1. New York, NY, USA: Association for Computing Machinery.
  11. Das, Devleena, and Sonia Chernova. 2020. "Leveraging Rationales to Improve Human Task Performance." In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI'20), Cagliari, Italy, 510-18. New York, NY, USA: Association for Computing Machinery.
  12. Davis, James H., 1992. "Some Compelling Intuitions About Group Consensus Decisions, Theoretical and Empirical Research, and Interpersonal Aggregation Phenomena: Selected Examples 1950- 1990." Organizational Behavior and Human Decision Processes 52(1): 3-38.
  13. Feng, Shi, and Jordan Boyd-Graber. 2019. "What Can AI Do for Me? Evaluating Machine Learning Interpretations in Cooperative Play." In Proceedings of the 24th International Conference on Intel- ligent User Interfaces (IUI'19), Marina del Ray, California, 229-39.
  14. Valeria González, Ana, Gagan Bansal, Angela Fan, Yashar Mehdad, Robin Jia, and Srinivasan Iyer. 2021. "Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations." In Findings of the Association for Compu- tational Linguistics: ACL-IJCNLP 2021, 1103-16. Association for Computational Linguistics.
  15. Richard Hackman, J., and Charles G Morris. 1975. "Group Tasks, Group Interaction Process, and Group Performance Effectiveness: A Review and Proposed Integration." Advances in Experimental Social Psychology 8: 45-99.
  16. Hill, Gayle W., 1982. "Group Versus Individual Performance: Are N + 1 Heads Better Than One?" Psychological Bulletin 91(3): 517-39.
  17. Horne, Benjamin D., Dorit Nevo, John O'Donovan, Jin-Hee Cho, and Sibel Adalı. 2019. "Rating Reliability and Bias in News Arti- cles: Does AI Assistance Help Everyone?" Proceedings of the International AAAI Conference on Web and Social Media 13(01): 247-56.
  18. Jacovi, Alon, and Yoav Goldberg. 2021. "Aligning Faithful Interpreta- tions with their Social Attribution." Transactions of the Association for Computational Linguistics 9: 294-310.
  19. Jesus, Sérgio, Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, and João Gama. 2021. "How Can I Choose An Explainer? An Application-grounded Eval- uation of Post-hoc Explanations." In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT'21, 805-15. New York, NY, USA: Association for Computing Machinery.
  20. Kim, Joon Sik, Valerie Chen, Danish Pruthi, Nihar B. Shah, and Ameet Talwalkar. 2023. "Assisting Human Decisions in Document Matching." arXiv preprint arXiv:2302.08450. · arXiv
  21. Kim, Sunnie S. Y., Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. 2022. "HIVE: Evaluating the Human Interpretability of Visual Explanations." In Computer Vision - ECCV 2022: 17th European Conference, Proceedings, Part XII, Tel Aviv, Israel, October 23-27, 2022, 280-98. Berlin, Heidelberg: Springer-Verlag.
  22. Kim, Sunnie S. Y., Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. ""Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction." In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI'23), Hamburg, Germany, 1-17.
  23. Laughlin, Patrick R., and John Adamopoulos. 1980. "Social Combina- tion Processes and Individual Learning for Six-Person Cooperative Groups on an Intellective Task." Journal of Personality and Social Psychology 38(6): 941-47.
  24. Laughlin, Patrick R., and Alan L Ellis. 1986. "Demonstrability and Social Combination Processes on Mathematical Intellective Tasks." Journal of Experimental Social Psychology 22(3): 177-89.
  25. Laughlin, Patrick R., and Homer H Johnson. 1966. "Group and Indi- vidual Performance on A Complementary Task as A Function of Initial Ability Level." Journal of Experimental Social Psychology 2(4): 407-14.
  26. Liu, Han, Vivian Lai, and Chenhao Tan. 2021. "Understanding the Effect of Out-of-distribution Examples and Interactive Explana- tions on Human-AI Decision Making." Proceedings of the ACM on Human-Computer Interaction 5(CSCW2): 1-45.
  27. Lombrozo, Tania. 2006. "The Structure and Function of Explana- tions." Trends in Cognitive Sciences 10(10): 464-70.
  28. Lundberg, Scott M., Bala Nair, Monica S. Vavilala, Mayumi Horibe, Michael J. Eisses, Trevor Adams, David E. Liston, et al. 2018. "Explainable Machine-Learning Predictions for the Prevention of Hypoxaemia During Surgery." Nature Biomedical Engineering 2(10): 749-60.
  29. Mittelstadt, Brent, Chris Russell, and Sandra Wachter. 2019. "Explaining Explanations in AI." In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*'19), Atlanta, GA, USA, 279-88. New York, NY, USA: Association for Computing Machinery.
  30. Nguyen, Giang, Daeyoung Kim, and Anh Nguyen. 2021. "The Effec- tiveness of Feature Attribution Methods and its Correlation with Automatic Evaluation Scores." Advances in Neural Information Processing Systems 34: 26422-36.
  31. Parrish, Alicia, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel Bowman. 2022. "Single- Turn Debate Does Not Help Humans Answer Hard Reading- Comprehension Questions." In Proceedings of the First Workshop on Learning with Natural Language Supervision, 17-28. Dublin, Ireland: Association for Computational Linguistics.
  32. Ribeiro Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016. ""Why Should I Trust You?": Explaining the Predictions of Any Classifier." In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'16, San Francisco, California, USA, 1135-44. New York, NY, USA: Association for Computing Machinery.
  33. Rudin, Cynthia. 2019. "Stop Explaining Black box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead." Nature Machine Intelligence 1(5): 206-15.
  34. Steiner, Ivan Dale. 1972. Group Process and Productivity. New York, NY, USA: Academic Press.
  35. Reza Taesiri, Mohammad, Giang Nguyen, and Anh Nguyen. 2022. "Visual Correspondence-Based Explanations Improve AI Robust- ness and Human-AI Team Accuracy." Advances in Neural Infor- mation Processing Systems 35: 34287-301.
  36. Tan, Chenhao. 2022. "On the Diversity and Limits of Human Expla- nations." In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, 2173-88. Seattle, USA: Association for Computational Linguistics.
  37. Tschandl, Philipp, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, et al. 2020. "Human-Computer Collaboration for Skin Cancer Recognition." Nature Medicine 26(8): 1229-34.
  38. Wang, Danding, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019. "Designing Theory-Driven User-Centric Explainable AI." In Proceedings of the 2019 CHI Conference on Human Factors in Com- puting Systems (CHI'19), Glasgow, Scotland UK, 1-15. New York, NY, USA: Association for Computing Machinery.
  39. Weerts, Hilde J. P., Werner van Ipenburg, and Mykola Pechenizkiy. 2019. "A Human-Grounded Evaluation of SHAP for Alert Process- ing." arXiv preprint arXiv:1907:03324.
  40. Zylberajch, Hugo, Piyawat Lertvittayakumjorn, and Francesca Toni. 2021. "HILDIF: Interactive Debugging of NLI Models Using Influence Functions." In Proceedings of the First Workshop on

Cited by

Sorted by citations

Authors
Fok, R., & Weld, D. S.
Affiliations
  • University of Washington (US)
  • Allen Institute for Artificial Intelligence (US)
Corresponding author
Fok, Raymond
ORCID
Venue
AI Magazine
Publisher
Wiley
Published
2024-07
Volume
45
Issue
3
Pages
317-332
ISSN
  • 0738-4602 (ISSN-L)
  • 2371-9621
DOI
10.1002/aaai.12182
OpenAlex
W4400195536
Cited by
81
References
34
Citations by year
2026: 27 · 2025: 23 · 2024: 26 · 2023: 5
FWCI
9.87
Type
journal article
Language
en
Open access
Licence
Locations
  • AI Magazine (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Office of Naval Research (N00014‐21‐1‐2707)
  • Allen Institute for Artificial Intelligence
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
explainable AI, AI explanations, verifiability, human-AI collaboration, interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review