CoChat

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Open access
Chen, Valerie, Liao, Q. Vera, Wortman Vaughan, Jennifer, Bansal, Gagan · 2023 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index; 44 of the 63 sources its claims cite were checked for support.

Abstract

AI explanations are often mentioned as a way to improve human-AI decision-making, but empirical studies have not found consistent evidence of explanations' effectiveness and, on the contrary, suggest that they can increase overreliance when the AI system is wrong. While many factors may affect reliance on AI support, one important factor is how decision-makers reconcile their own intuition---beliefs or heuristics, based on prior knowledge, experience, or pattern recognition, used to make judgments---with the information provided by the AI system to determine when to override AI predictions. We conduct a think-aloud, mixed-methods study with two explanation types (feature- and example-based) for two prediction tasks to explore how decision-makers' intuition affects their use of AI predictions and explanations, and ultimately their choice of when to rely on AI. Our results identify three types of intuition involved in reasoning about AI predictions and explanations: intuition about the task outcome, features, and AI limitations. Building on these, we summarize three observed pathways for decision-makers to apply their own intuition and override AI predictions. We use these pathways to explain why (1) the feature-based explanations we used did not improve participants' decision outcomes and increased their overreliance on AI, and (2) the example-based explanations we used improved decision-makers' performance over feature-based explanations and helped achieve complementary human-AI performance. Overall, our work identifies directions for further development of AI decision-support systems and explanation methods that help decision-makers effectively apply their intuition to achieve appropriate reliance on AI.

Overview

CoChat Summary

TL;DR

This think-aloud mixed-methods study (N=26) examines how decision-makers’ intuition shapes reliance on AI predictions with feature- versus example-based explanations across income prediction and biography classification. It identifies three intuition-driven pathways to override AI and finds example-based explanations supported complementary human-AI performance, while feature-based explanations did not improve outcomes and increased overreliance on incorrect AI.

Key findings

  • The paper identifies three types of intuition involved in reasoning about AI predictions and explanations—intuition about the task outcome, features, and AI limitations—and summarizes three pathways to override AI: strong outcome intuition disagreeing with AI, applying feature intuition to discredit the explanation, and recognizing signals of prediction unreliability.
  • Example-based explanations led to complementary performance: income prediction accuracy was 61.1% ± 2.9% with no AI, 60.6% ± 4.2% with feature-based explanations, and 71.1% ± 2.9% with example-based explanations; biography classification was 60.0% ± 4.4%, 64.4% ± 4.3%, and 71.2% ± 4.6%, respectively, while the AI alone achieved 62.5% on the presented instances.
  • Feature-based explanations did not significantly improve overall accuracy over No AI (SE = 0.00, p = 0.88 for income prediction; SE = 0.04, p = 0.51 for biography classification) and decreased accuracy when the AI was incorrect (SE = −0.23, p = 0.09 for income prediction; SE = −0.24, p < 0.01 for biography classification); example-based explanations had no such negative effect and were significant for biography classification (p = 0.04) and marginally significant for income prediction (p = 0.07).
  • Subjective preferences diverged from performance: 11 participants preferred feature-based explanations, 10 preferred example-based, and 5 were impartial; 7 of the 11 who preferred feature-based explanations performed better with example-based explanations. Among the 5 participants with lowest domain knowledge (average phase 2 accuracy = 43.8%), example-based explanations improved accuracy to 70.0% while feature-based explanations yielded 52.5%.

Methods

  • Mixed-methods think-aloud study with N=26 participants, recruited via Twitter and internal message boards at a large international technology company, limited to US-based participants and selected to diversify ML and XAI backgrounds; within-subjects design where each participant was randomly assigned to one of two tasks but engaged with both explanation types in randomized order.
  • Two prediction tasks: income prediction (>$50,000 from Folktables 2018 US Census data, random forest with 80.8% held-out accuracy) and biography classification (five professions from the BIOS dataset, bag-of-words random forest with 75.4% held-out accuracy). Phase 2 had 16 instances with no AI; Phase 3 showed the same 16 instances with AI support, 8 per explanation type, sampled as 5 AI-correct and 3 AI-incorrect per explanation type, with think-aloud during Phase 3 and a post-study interview.
  • Analysis combined exploratory mixed-effects regression on decision accuracy with participants as a random effect and explanation type as a fixed effect (No AI as reference), plus qualitative open coding informed by Grounded Theory on think-aloud and interview data, with coders initially blind to decision correctness.

Limitations

  • The paper states that think-aloud may not resemble realistic decision-making and may increase cognitive engagement; think-aloud data are observational and cannot isolate the effect of different intuition types or identify precise causes; the two-phase repeated-instance design may strengthen prior intuition; and the study is limited by small sample size, low-stakes tasks, use of an unfaithful post-hoc explanation method (LIME), and a sample skewed toward highly educated and ML-experienced individuals, so the identified intuition types may not be complete or fully generalizable.
  • Evident from the text: the convenience sample from a technology company and Twitter, restricted to US participants, further limits generalizability; the paper itself cautions that the quantitative analysis is exploratory and that p-values should be interpreted with caution given the small sample size.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3610219

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

18 Valid · 5 Not valid · 8 Not checked · 179 Own results and interpretations

101 more claims · 1 with their cited sources checked

Create a free account to see all 210 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 52
  • It may state two separate claims as one. 63
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 1

  1. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning. PMLR, 2668--2677.

Cited by

See all 220

Metadata

Authors
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G.
Affiliations
  • Carnegie Mellon University (US)
  • Microsoft Research Montréal (Canada) (CA)
  • Microsoft Research, New York, NY, USA
  • Microsoft (United States) (US)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2023-09-28
Volume
7
Issue
CSCW2
Pages
1-32
ISSN
2573-0142 (ISSN-L)
OpenAlex
W4387344906
Cited by
220
References
74
Citations by year
2026: 90 · 2025: 72 · 2024: 47 · 2023: 10
FWCI
15.59
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Microsoft Research
  • Institut National Du Cancer
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
human intuition, AI explanations, human-AI decision-making, overreliance on AI, feature-based explanations, example-based explanations, human-AI complementarity, decision support system, think-aloud

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.