Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Open accessThe DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index; 44 of the 63 sources its claims cite were checked for support.
Abstract
AI explanations are often mentioned as a way to improve human-AI decision-making, but empirical studies have not found consistent evidence of explanations' effectiveness and, on the contrary, suggest that they can increase overreliance when the AI system is wrong. While many factors may affect reliance on AI support, one important factor is how decision-makers reconcile their own intuition---beliefs or heuristics, based on prior knowledge, experience, or pattern recognition, used to make judgments---with the information provided by the AI system to determine when to override AI predictions. We conduct a think-aloud, mixed-methods study with two explanation types (feature- and example-based) for two prediction tasks to explore how decision-makers' intuition affects their use of AI predictions and explanations, and ultimately their choice of when to rely on AI. Our results identify three types of intuition involved in reasoning about AI predictions and explanations: intuition about the task outcome, features, and AI limitations. Building on these, we summarize three observed pathways for decision-makers to apply their own intuition and override AI predictions. We use these pathways to explain why (1) the feature-based explanations we used did not improve participants' decision outcomes and increased their overreliance on AI, and (2) the example-based explanations we used improved decision-makers' performance over feature-based explanations and helped achieve complementary human-AI performance. Overall, our work identifies directions for further development of AI decision-support systems and explanation methods that help decision-makers effectively apply their intuition to achieve appropriate reliance on AI.
Overview
CoChat Summary
TL;DR
This think-aloud mixed-methods study (N=26) examines how decision-makers’ intuition shapes reliance on AI predictions with feature- versus example-based explanations across income prediction and biography classification. It identifies three intuition-driven pathways to override AI and finds example-based explanations supported complementary human-AI performance, while feature-based explanations did not improve outcomes and increased overreliance on incorrect AI.
Key findings
- The paper identifies three types of intuition involved in reasoning about AI predictions and explanations—intuition about the task outcome, features, and AI limitations—and summarizes three pathways to override AI: strong outcome intuition disagreeing with AI, applying feature intuition to discredit the explanation, and recognizing signals of prediction unreliability.
- Example-based explanations led to complementary performance: income prediction accuracy was 61.1% ± 2.9% with no AI, 60.6% ± 4.2% with feature-based explanations, and 71.1% ± 2.9% with example-based explanations; biography classification was 60.0% ± 4.4%, 64.4% ± 4.3%, and 71.2% ± 4.6%, respectively, while the AI alone achieved 62.5% on the presented instances.
- Feature-based explanations did not significantly improve overall accuracy over No AI (SE = 0.00, p = 0.88 for income prediction; SE = 0.04, p = 0.51 for biography classification) and decreased accuracy when the AI was incorrect (SE = −0.23, p = 0.09 for income prediction; SE = −0.24, p < 0.01 for biography classification); example-based explanations had no such negative effect and were significant for biography classification (p = 0.04) and marginally significant for income prediction (p = 0.07).
- Subjective preferences diverged from performance: 11 participants preferred feature-based explanations, 10 preferred example-based, and 5 were impartial; 7 of the 11 who preferred feature-based explanations performed better with example-based explanations. Among the 5 participants with lowest domain knowledge (average phase 2 accuracy = 43.8%), example-based explanations improved accuracy to 70.0% while feature-based explanations yielded 52.5%.
Methods
- Mixed-methods think-aloud study with N=26 participants, recruited via Twitter and internal message boards at a large international technology company, limited to US-based participants and selected to diversify ML and XAI backgrounds; within-subjects design where each participant was randomly assigned to one of two tasks but engaged with both explanation types in randomized order.
- Two prediction tasks: income prediction (>$50,000 from Folktables 2018 US Census data, random forest with 80.8% held-out accuracy) and biography classification (five professions from the BIOS dataset, bag-of-words random forest with 75.4% held-out accuracy). Phase 2 had 16 instances with no AI; Phase 3 showed the same 16 instances with AI support, 8 per explanation type, sampled as 5 AI-correct and 3 AI-incorrect per explanation type, with think-aloud during Phase 3 and a post-study interview.
- Analysis combined exploratory mixed-effects regression on decision accuracy with participants as a random effect and explanation type as a fixed effect (No AI as reference), plus qualitative open coding informed by Grounded Theory on think-aloud and interview data, with coders initially blind to decision correctness.
Limitations
- The paper states that think-aloud may not resemble realistic decision-making and may increase cognitive engagement; think-aloud data are observational and cannot isolate the effect of different intuition types or identify precise causes; the two-phase repeated-instance design may strengthen prior intuition; and the study is limited by small sample size, low-stakes tasks, use of an unfaithful post-hoc explanation method (LIME), and a sample skewed toward highly educated and ML-experienced individuals, so the identified intuition types may not be complete or fully generalizable.
- Evident from the text: the convenience sample from a technology company and Twitter, restricted to US participants, further limits generalizability; the paper itself cautions that the quantitative analysis is exploratory and that p-values should be interpreted with caution given the small sample size.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Valid · 5 Not valid · 8 Not checked · 179 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Our exploratory quantitative analysis shows that, consistent with prior work [4,82,87], featurebased explanations did not improve participants' decision outcomes compared to their performance without AI support, and led to overreliance on incorrect AI predictions.”
- Also stated in: § Abstract · plain
“We use these pathways to explain why (1) the feature-based explanations we used did not improve participants' decision outcomes and increased their overreliance on AI”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 1 INTRODUCTION
What the sources say
- Disputedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. sourceExperimental — accuracy not yet measured
Section 5.1 (p. 8) says all team conditions, including explanation conditions, achieved complementary performance—exceeding unassisted human performance. The paper's finding of no significant improvement concerns explanations compared with AI assistance that showed confidence scores, not compared with no AI support (Section 5.1, p. 7; Conclusions, p. 14). (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Partly supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In §4.2.2 Results (page unavailable), the paper reports no significant improvement in AI-assisted accuracy when explanations were shown. But its comparison is across AI-information conditions, not against participants’ performance without AI support; it reports participants’ unaided accuracy separately as 63%. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“On the contrary, several studies found that showing feature-based explanations increases people's tendency to over-rely on the model when it is wrong, compared to showing only the AI predictions [4,68,82,87].”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 2.2.1 Can XAI methods improve appropriate reliance?
What the sources say
- Supportedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. source
The pilot study (Section 3.2, p. 4) compared feature-based explanations with a condition showing only the model prediction and confidence. It reports increased reliance on incorrect recommendations when the predicted class was explained. The main study (Section 5.1, pp. 8–9) found the effect varied by dataset, but the pilot directly supports the claim attributed to this paper. (LLM, conf=0.91)
- Partly supportedcites [68] 68. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1-52. sourceExperimental — accuracy not yet measured
The Overview of experiments (§1.2; page not available) reports that participants shown a clear model were less able to correct sizable mistakes on unusual apartments than participants shown a black-box model. But this paper compares access to model internals, not a distinct feature-based explanation alongside an otherwise identical prediction-only display; §5.2 also reports that the effect did not appear in the third experiment. The claim therefore captures a finding but describes the intervention and consistency more broadly than the source supports. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Disputedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In §4.2 Results (page unavailable), the paper compares feature-based explanations with a prediction-only baseline and finds no significant difference in participants’ willingness to switch to the AI’s prediction (p = .66). It concludes that explanations did not perceptibly affect trust, contrary to the claimed increase in over-reliance when the model was wrong. The paper discusses increased acceptance of incorrect predictions in another study, but does not find that effect in its own comparison. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 2.2 Human-AI Decision-Making
What the sources say
- Partly supportedcites [6] 6. Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M Vardoulakis. 2020. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1-12. sourceExperimental — accuracy not yet measured
The “Challenges evaluating AI in clinical settings” section reports that mammography computer-aided detection generally did not improve radiologists’ diagnostic accuracy. That supports an example of human–AI assistance failing to improve performance, but this paper does not establish the broader claim that many empirical studies found no complementary performance, nor does it report a comparative performance result for its own deployment. Page numbers were not available in our copy. (LLM, conf=0.76)
- Supportedcites [16] 16. Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2020. Feature-Based Explanations Don't Help People Detect Misclas- sifications of Online Toxicity. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 95-106. sourceExperimental — accuracy not yet measured
In Accuracy and Agreement, the paper reports that showing participants the model’s prediction slightly reduced accuracy, while explanations did not significantly change that result. Experiment Design for Evaluating Interpretability also describes a gap between hoped-for and demonstrated gains in human–model performance across this and similar studies. The paper therefore supports the claim as one example among the studies cited, rather than as evidence about every human–AI decision task. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [37] 37. Ben Green and Yiling Chen. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-24. source
- Supportedcites [52] 52. Vivian Lai, Han Liu, and Chenhao Tan. 2020. " Why is' Chicago'deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13. sourceExperimental — accuracy not yet measured
The Results (p. 8) show that human participants rarely outperformed the ML model: 0.4% in Experiment 1 and 8.1% with effective real-time assistance in Experiment 2. This study therefore supports the cited sentence as one example of limited complementary performance, while acknowledging that a minority of participants did outperform the model. (LLM, conf=0.88)
- Supportedcites [58] 58. Han Liu, Vivian Lai, and Chenhao Tan. 2021. Understanding the effect of out-of-distribution examples and interactive explanations on human-AI decision making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-45. sourceExperimental — accuracy not yet measured
Section 2.1 reports that, across many studies of challenging prediction tasks, human–AI teams seldom outperform AI alone; it also identifies a notable exception. Because the paper defines complementary performance as outperforming both AI alone and humans alone, those reported studies support the citing sentence’s claim. (LLM, conf=0.91)
- Supportedcites [61] 61. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749-760. source
In the Results section, the paper reports that anaesthesiologists assisted by Prescience performed better than unaided anaesthesiologists but slightly worse than Prescience alone. It therefore supports this citation as one example of a study that did not observe complementary performance; it does not itself establish how many studies found that result. (LLM, conf=0.90)
- Supportedcites [68] 68. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1-52. source
This paper supports its use as one example in the citing sentence’s list of empirical studies. Section 3.2 (page unavailable) reports that, for typical apartments, participants using a model made less accurate predictions than the model alone, although they did better than participants without a model. The paper does not itself establish how many studies found this pattern. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Not checked — no full text of the sourcecites [84] 84. Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. 2020. How do visual explanations foster end users' appropriate trust in machine learning?. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 189-201. source
- Supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In 3.2 Results (page not provided), the paper reports that human–AI joint decisions were not more accurate than AI decisions alone, even in the low-confidence cases where the authors expected a benefit. This study therefore supports the cited sentence as one example of an empirical study that did not observe complementary performance. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our exploratory quantitative analysis shows that, consistent with prior work [4,82,87], featurebased explanations did not improve participants' decision outcomes compared to their performance without AI support, and led to overreliance on incorrect AI predictions.”
- Also stated in: § Abstract · plain
“increased their overreliance on AI”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 1 INTRODUCTION
What the sources say
- Supportedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. source
Section 3.1 (p. 4) describes the study’s feature-based explanations as highlights showing the importance of input words. The pilot-study findings (Section 3.2, p. 4) report increased reliance on AI recommendations even when they were incorrect; Section 5.1 (p. 8) reports a similar effect for explanations of the top prediction, though the effect varied by task. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Partly supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In 4.2 Results (page unavailable), the paper reports that local feature-based explanations did not improve trust calibration or AI-assisted accuracy. It does not show that they caused greater reliance on incorrect predictions: agreement was similar with and without explanations. Its discussion of increased acceptance of incorrect predictions concerns Lai and Tan’s study, not its own experiment. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“This suggests that example-based explanations can provide additional context and support inductive reasoning [11] to help people form additional intuition about features.”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > Applying intuition about features to reason about explanations.
What the sources say
- Supportedcites [11] 11. Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the 25th international conference on intelligent user interfaces. 454-464. source
Section 3.1 (p. 3) describes example-based explanations that show four additional food images and ask participants to recognize ingredients contributing to fat content and draw their own conclusions. It explicitly identifies this as inductive reasoning, supporting the paraphrase about context and intuition about features. (LLM, conf=0.91)
hedgedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
101 more claims · 1 with their cited sources checked
Create a free account to see all 210 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 52
- It may state two separate claims as one. 63
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings suggest that a reflective approach toward the use of GenAI tools, which can lead to what prior work refers to as”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conversely, with the important caveat that users' self-confidence is a subjective measure of their knowledge, experiences, and abilities on the tasks [20,59,85], higher self-confidence is associated with more critical thinking, even though workers who are confident in their own skills tend to perceive greater effort in these tasks, particularly when evaluating and applying AI responses.”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This can lead to disastrous outcomes through overreliance [19, 25, 83, 102], when people take actions based on incorrect outputs.”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vice versa, people may -and often do -trust an untrustworthy model (Chen, Liao, Vaughan, & Bansal, 2023).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Chen, Liao, Vaughan, & Bansal, 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They show a number of studies that suggest that explanations can increase users' reliance on AI predictions even if the AI system's predictions are wrong.”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 4.3.1 Subjective preferences of explanation types.
“For example, P17 said "when the data was super consistent, I was gonna go with it"”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 4.3.1 Subjective preferences of explanation types.
“"when the data was super consistent, I was gonna go with it" (P32).”
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Explanations can also be classified as global or local [1, 35].”
BackgroundMethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“First, we overview related work on explainable AI (XAI), focusing on XAI for ML models—what is sometimes referred to as interpretability in the ML literature [26, 53].”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Research works printed but not linked: 1
- Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning. PMLR, 2668--2677.
Cited by
Items 101–154 of 154
Exploring Creator-Centric Methods for LLM-Assisted Interactive Storytelling
Li, Y. F., Wu, S., Lujin, Z., Guo, Z., Lu, W., & Yip, D. K. M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791362 · Cited by 3
Does Sycophancy Change Decisions? Effect of LLM Sycophancy on AI-Assisted Decision-Making
Li, Z., Pan, J., Liu, Q., Xi, Y., Zhou, Y., Jin, Y., Mao, R., & Chen, P. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790934 · Cited by 13
A metacognitive approach to learning and performance in human-AI interaction
Fan, Y., Yan, L., Molenaar, I., Greiff, S., & Gašević, D. (2026) · Communications Psychology · doi:10.1038/s44271-026-00548-y · Cited by 1
AI Integration for Decision Making
Raturi, V., Yadav, A., Kumar, S., Akram, S. V., & Chanti, Y. (2023) · International Conference on Advancement in Electronics & Communication Engineering (AECE) · doi:10.1109/aece59614.2023.10428584
“This variant helps to go for a suitable decision and also learns the repetitive steps [19], [20].”
BackgroundChanges Accompanied by Artificial Intelligence (AI) and How Organizations Respond to These Changes
Farhan, A. F., & Hatem, G. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1007615 · Cited by 4
Robust Relatable Explanations of Machine Learning with Disentangled Cue-specific Saliency
Abichandani, H. S., Zhang, W., & Lim, B. Y. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712105 · Cited by 1
Synergizing next-generation data science and blockchain: catalysts for Society 5.0 implementation
Afroz, M., & Nyakwende, E. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-33498-6.00010-8
The Role of Artificial Intelligence in Backing Up Human Insight: A Powerful Partnership
Kim, H., & Lee, H. S. (2025) · MechEcology · doi:10.23104/me.e7 · Cited by 1
Centaurs and Cyborgs: Models of Collaboration
(2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-2518-7.ch003
Human-Centered and Contextual Assessment of Human-AI Decision-Making Interventions
Nasr, S., & Johnson, D. S. (2025) · Contextualizing Explanations · doi:10.64136/mznt1005
Sotnikova, S. I. (2025) · Vestnik NSUEM · doi:10.34020/2073-6495-2025-4-010-025
Sawada, S., & Toyoda, M. (2025) · International Symposium on Visual Information Communication and Interaction (VINCI) · doi:10.1145/3769534.3769602
Overreliance on AI in Oil and Gas Cybersecurity: A Review of Risks and Mitigation Strategies
Alkhalaf, D., Shareef, R., Almousa, S., & Alshamrani, S. (2026) · IPTC Summit on AI for the Energy Industry · doi:10.2523/iptc-25174-ms · Cited by 1
Applications of Hybrid and Explainable AI Models for Public Employment ServicesPreprint
Kooistra, J., Machado, M., Misheva, B. H., & Moreira, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6115035
Educators’ Evaluation of Explanation Types in XAI for Higher Education Dropout
Maathuis, H., Glazenborg, J., Grol, M. J., & Quille, K. (2026) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3777490.3777499
HUMAN–AI CO-CREATION MODELS IN CONCEPTUAL ART
Bhise, S., Pradhan, I., Keerthi, H., Arsalwad, G. P., Tolani, K., & Saraswati, B. (2026) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v7.i1s.2026.7121
Balaskas, S., Komis, K., Yfantidou, I., & Skandali, D. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10030022 · Cited by 3
Passing the Buck to AI: How Individuals’ Decision-Making Patterns Affect Reliance on AI
Mei, K. X., Pang, R. Y., Lyford, A., Wang, L. L., & Reinecke, K. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3786326 · Cited by 8
“This section also sets the stage for our subsequent exploration of factors that have been found to influence human-AI interaction—in particular trust and reliance—and how decision-making patterns may be a missing, but important piece in the puzzle.”
Bertrand, A. (2024) · theses.fr (ABES) · doi:10.70675/216d3647z78bcz4c68z9ba9z2d0a029cf1b4 · Cited by 8
Social and Emotional Uses of AI
Tseng, E., Adler, D. A., Walker, A. M., Shelby, R. M., Chancellor, S., Kim, E., Pendse, S. R., & Zhang, R. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778699
Detecting Cross-Area Concept Links in HCI Using Persistent Homology
Shimizu, K., Komatsu, S., Hayashi, K., et al. (2026) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798751
Popowski, L., Vasconcelos, H., Heinsen, I. J. F., Mgbahurike, C., Herbrich, R., Hancock, J. T., & Bernstein, M. S. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788067 · Cited by 1
Offline Reinforcement Learning for Adaptive Support in AI-Assisted Decision-Making
Buçinca, Z., Swaroop, S., Paluch, A. E., Murphy, S. A., & Gajos, K. Z. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3815418 · Cited by 54
“However, mounting evidence shows that decision-makers struggle to incorporate AI recommendations into their decisions, often either over-relying or under-relying on AI, even when explanations are provided [5, 8, 13, 23, 43, 76, 80, 88, 91, 98, 98, 117].”
InfluentialFredriksdotter, K., Kuratomi, A., & Quintero, L. (2026) · doi:10.2196/preprints.104490
Madhavan, M. M., Satyapriya, Sahu, S., Velayudhan, P. K., Smitha, S. C., Verma, A. P., Pawariya, V., Barua, S., Mahra, G. S., Bishnoi, S., et al. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1769064 · Cited by 1
“In this context, recent studies have examined LLMs for forecasting and behavioral prediction tasks (20–22) and emerging evidence suggests that LLM-based systems may emulate aspects of human behavioral responses in experimental settings (23, 24).”
Preaching Meaning, Practicing the Motions: A Critical Analysis of XAI Research
van der Linden, I., & Ghebreab, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812356
Reliance-Aware Design of Interactive Explainable User Interfaces
Filho, J. C. D. S., Belloum, R., Monteiro, I. T., & de Oliveira, K. M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812231 · Cited by 2
“XAI systems have been widely used to support decision-making processes in several fields, such as medical diagnosis [50, 52], loan approval [55], criminal recidivism [42, 74], education [10], and entertainment [39, 69].”
Resume Screening, Fast and Slow: (Biased) AI Recommendations' Influence on Human Decision Making
Wilson, K., Sim, M., Gueorguieva, A.-M., Chatterjee, S., & Caliskan, A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812381 · Cited by 1
“Furthermore, a number of studies have also found that the effectiveness of AI-assisted decision making can be influenced by the activation of System 1 or 2 thinking and find that over-reliance is more likely when the latter is not activated [10, 14, 56].”
Impact of AI Explanation Styles on IT Students’ Debugging Accuracy, Confidence, and Reliance
Dzvapatsva, G. P., Ncube, P. D. N., Chinhamo, E., & Matobobo, C. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574352
“Empirical findings in human-AI interaction consistently show that explanation design shapes how users interpret system competence, allocate cognitive effort, and decide when to rely on AI recommendations [25], [31], [32].”
Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making
Spillner, L., Rockstroh, J., Wenig, N., Zargham, N., Kiefner, L., Porzel, R., & Malaka, R. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816231 · Cited by 1
How Information Detail and Confidence in AI Responses Shape User Beliefs
Wu, Z., Jobanputra, M., Demberg, V., Hullman, J., & Feit, A. M. (2026) · International Conference on Human-AI Interaction and Experience Design (HAXD) · doi:10.1109/haxd70072.2026.11620844
“Prior work shows that, even when an AI takes the same stance as a user, the specific content of the response, such as the use of explanations [3], [15]±[17], the inclusion of citations [3], [18], or offering personalized suggestions [19], can change a user’s decisions.”
Ching-Tsung, J., Lu, W., & Sang, T. Q. (2026) · Research Square · doi:10.21203/rs.3.rs-10421674/v1
Cao, T. T., & Tuân, N. P. (2026) · Journal of Internet Commerce · doi:10.1080/15332861.2026.2711973 · Cited by 2
Not All Trust Is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Making
Spillner, L., Ringe, R., Porzel, R., & Malaka, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_15 · Cited by 5
EvidenceTrail: Auditable Evidence Graphs for Process-Excursion Decision SupportPreprint
hu, Q. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7292207
Imperial, R. M. A., & Artiola, A. S. (2026) · Psychology and Education A Multidisciplinary Journal · doi:10.70838/pemj.610409
Boyacı, T., Canyakmaz, C., & de Véricourt, F. (2026) · Management Science · doi:10.1287/mnsc.2024.07450 · Cited by 4
Digital empowerment-oriented intelligent interaction mechanism design
Dong, S. (2026) · International Conference on Innovation Management and Information Systems (ICIMIS) · doi:10.1145/3821966.3822040
Tuo, B., Miller, T., & Demartini, G. (2026) · ACM Conference on Human-AI Complementarity and Alignment · doi:10.1145/3834580.3838739
Li, Z., Ma, Z., He, S., Lu, Z., Xiao, Z., & Yin, M. (2026) · ACM Conference on Human-AI Complementarity and Alignment · doi:10.1145/3834580.3838738
Fernández, R. C., & Solórzano, M. (2026) · Applied Artificial Intelligence · doi:10.1080/08839514.2026.2726917
Conformal Inverse Optimization for Adherence-Aware Prescriptive Analytics
Chan, T. C. Y., Delage, E., & Lin, B. (2026) · Operations Research · doi:10.1287/opre.2024.1376 · Cited by 4
From Potential to Reality: Gaps in Realizing Human–AI Complementarity in Educational Tasks
Jiang, B., Wang, J., Lu, Y., Liu, J., & Jiang, Y. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2732482
Tuo, B., Wang, B. T., Miller, T., & Demartini, G. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3848504
“Researchers have proposed AI as debate partners, counter-argument generators, hypothesis reviewers, and deliberative facilitators — roles designed to structure reasoning by improving argument quality, balancing participation, and ensuring fairness of influence [15, 19, 50, 51].”
Empowered yet unethical? Understanding GenAI misuse intention in generative AI chatbot use
Hung, S., & Jian, J.-H. (2026) · Asia Pacific Journal of Marketing and Logistics · doi:10.1108/apjml-04-2026-0774
Wired to believe: source-authority cues shape perceived AI capability and neural persuasion
Ding, Y., Deng, Z., Zhang, W., Lyu, W., Duffy, B. M., & Zhang, D. (2026) · Ergonomics · doi:10.1080/00140139.2026.2734930 · Cited by 2
Human and AI in Decision Making: A Systematic Review of Trust, Fairness, and Empathy
Arora, A., & Arora, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9599-9.ch015
Dorsch, J., Moll, M., & Deroy, O. D. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01166-6 · Cited by 1
“This positive effect is amplified when reliability scores are revealed only after the human user has made a preliminary judgment (Chen et al., 2023).”
Lee, J., Bae, J., Ryu, E. S., Kim, L., Kim, D., Lee, S., & Yun, M. H. (2026) · Ergonomics · doi:10.1080/00140139.2026.2731219
Ungureanu, V. (2026) · AI & Society · doi:10.1007/s00146-026-03352-8
Ortegón, A. D. P. B., Hernandez, P. R. J., & Ricaurte, G. F. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1488-2.ch002
From Oracles to Productive Friction: Designing for Meaningful Human Oversight in Clinical AI
Rubegni, E., Natali, C., Valsecchi, C., Rizzo, S., Ayoub, O., & Facchini, A. (2026) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v9i3.43300
Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.21144
Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.11179
Metadata
- Authors
- Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G.
- Affiliations
- Carnegie Mellon University (US)
- Microsoft Research Montréal (Canada) (CA)
- Microsoft Research, New York, NY, USA
- Microsoft (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-28
- Volume
- 7
- Issue
- CSCW2
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4387344906
- Cited by
- 220
- References
- 74
- Citations by year
- 2026: 90 · 2025: 72 · 2024: 47 · 2023: 10
- FWCI
- 15.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Institut National Du Cancer
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human intuition, AI explanations, human-AI decision-making, overreliance on AI, feature-based explanations, example-based explanations, human-AI complementarity, decision support system, think-aloud
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.