Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Open accessThe DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index; 44 of the 63 sources its claims cite were checked for support.
Abstract
AI explanations are often mentioned as a way to improve human-AI decision-making, but empirical studies have not found consistent evidence of explanations' effectiveness and, on the contrary, suggest that they can increase overreliance when the AI system is wrong. While many factors may affect reliance on AI support, one important factor is how decision-makers reconcile their own intuition---beliefs or heuristics, based on prior knowledge, experience, or pattern recognition, used to make judgments---with the information provided by the AI system to determine when to override AI predictions. We conduct a think-aloud, mixed-methods study with two explanation types (feature- and example-based) for two prediction tasks to explore how decision-makers' intuition affects their use of AI predictions and explanations, and ultimately their choice of when to rely on AI. Our results identify three types of intuition involved in reasoning about AI predictions and explanations: intuition about the task outcome, features, and AI limitations. Building on these, we summarize three observed pathways for decision-makers to apply their own intuition and override AI predictions. We use these pathways to explain why (1) the feature-based explanations we used did not improve participants' decision outcomes and increased their overreliance on AI, and (2) the example-based explanations we used improved decision-makers' performance over feature-based explanations and helped achieve complementary human-AI performance. Overall, our work identifies directions for further development of AI decision-support systems and explanation methods that help decision-makers effectively apply their intuition to achieve appropriate reliance on AI.
Overview
CoChat Summary
TL;DR
This think-aloud mixed-methods study (N=26) examines how decision-makers’ intuition shapes reliance on AI predictions with feature- versus example-based explanations across income prediction and biography classification. It identifies three intuition-driven pathways to override AI and finds example-based explanations supported complementary human-AI performance, while feature-based explanations did not improve outcomes and increased overreliance on incorrect AI.
Key findings
- The paper identifies three types of intuition involved in reasoning about AI predictions and explanations—intuition about the task outcome, features, and AI limitations—and summarizes three pathways to override AI: strong outcome intuition disagreeing with AI, applying feature intuition to discredit the explanation, and recognizing signals of prediction unreliability.
- Example-based explanations led to complementary performance: income prediction accuracy was 61.1% ± 2.9% with no AI, 60.6% ± 4.2% with feature-based explanations, and 71.1% ± 2.9% with example-based explanations; biography classification was 60.0% ± 4.4%, 64.4% ± 4.3%, and 71.2% ± 4.6%, respectively, while the AI alone achieved 62.5% on the presented instances.
- Feature-based explanations did not significantly improve overall accuracy over No AI (SE = 0.00, p = 0.88 for income prediction; SE = 0.04, p = 0.51 for biography classification) and decreased accuracy when the AI was incorrect (SE = −0.23, p = 0.09 for income prediction; SE = −0.24, p < 0.01 for biography classification); example-based explanations had no such negative effect and were significant for biography classification (p = 0.04) and marginally significant for income prediction (p = 0.07).
- Subjective preferences diverged from performance: 11 participants preferred feature-based explanations, 10 preferred example-based, and 5 were impartial; 7 of the 11 who preferred feature-based explanations performed better with example-based explanations. Among the 5 participants with lowest domain knowledge (average phase 2 accuracy = 43.8%), example-based explanations improved accuracy to 70.0% while feature-based explanations yielded 52.5%.
Methods
- Mixed-methods think-aloud study with N=26 participants, recruited via Twitter and internal message boards at a large international technology company, limited to US-based participants and selected to diversify ML and XAI backgrounds; within-subjects design where each participant was randomly assigned to one of two tasks but engaged with both explanation types in randomized order.
- Two prediction tasks: income prediction (>$50,000 from Folktables 2018 US Census data, random forest with 80.8% held-out accuracy) and biography classification (five professions from the BIOS dataset, bag-of-words random forest with 75.4% held-out accuracy). Phase 2 had 16 instances with no AI; Phase 3 showed the same 16 instances with AI support, 8 per explanation type, sampled as 5 AI-correct and 3 AI-incorrect per explanation type, with think-aloud during Phase 3 and a post-study interview.
- Analysis combined exploratory mixed-effects regression on decision accuracy with participants as a random effect and explanation type as a fixed effect (No AI as reference), plus qualitative open coding informed by Grounded Theory on think-aloud and interview data, with coders initially blind to decision correctness.
Limitations
- The paper states that think-aloud may not resemble realistic decision-making and may increase cognitive engagement; think-aloud data are observational and cannot isolate the effect of different intuition types or identify precise causes; the two-phase repeated-instance design may strengthen prior intuition; and the study is limited by small sample size, low-stakes tasks, use of an unfaithful post-hoc explanation method (LIME), and a sample skewed toward highly educated and ML-experienced individuals, so the identified intuition types may not be complete or fully generalizable.
- Evident from the text: the convenience sample from a technology company and Twitter, restricted to US participants, further limits generalizability; the paper itself cautions that the quantitative analysis is exploratory and that p-values should be interpreted with caution given the small sample size.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Valid · 5 Not valid · 8 Not checked · 179 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Our exploratory quantitative analysis shows that, consistent with prior work [4,82,87], featurebased explanations did not improve participants' decision outcomes compared to their performance without AI support, and led to overreliance on incorrect AI predictions.”
- Also stated in: § Abstract · plain
“We use these pathways to explain why (1) the feature-based explanations we used did not improve participants' decision outcomes and increased their overreliance on AI”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 1 INTRODUCTION
What the sources say
- Disputedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. sourceExperimental — accuracy not yet measured
Section 5.1 (p. 8) says all team conditions, including explanation conditions, achieved complementary performance—exceeding unassisted human performance. The paper's finding of no significant improvement concerns explanations compared with AI assistance that showed confidence scores, not compared with no AI support (Section 5.1, p. 7; Conclusions, p. 14). (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Partly supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In §4.2.2 Results (page unavailable), the paper reports no significant improvement in AI-assisted accuracy when explanations were shown. But its comparison is across AI-information conditions, not against participants’ performance without AI support; it reports participants’ unaided accuracy separately as 63%. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“On the contrary, several studies found that showing feature-based explanations increases people's tendency to over-rely on the model when it is wrong, compared to showing only the AI predictions [4,68,82,87].”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 2.2.1 Can XAI methods improve appropriate reliance?
What the sources say
- Supportedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. source
The pilot study (Section 3.2, p. 4) compared feature-based explanations with a condition showing only the model prediction and confidence. It reports increased reliance on incorrect recommendations when the predicted class was explained. The main study (Section 5.1, pp. 8–9) found the effect varied by dataset, but the pilot directly supports the claim attributed to this paper. (LLM, conf=0.91)
- Partly supportedcites [68] 68. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1-52. sourceExperimental — accuracy not yet measured
The Overview of experiments (§1.2; page not available) reports that participants shown a clear model were less able to correct sizable mistakes on unusual apartments than participants shown a black-box model. But this paper compares access to model internals, not a distinct feature-based explanation alongside an otherwise identical prediction-only display; §5.2 also reports that the effect did not appear in the third experiment. The claim therefore captures a finding but describes the intervention and consistency more broadly than the source supports. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Disputedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In §4.2 Results (page unavailable), the paper compares feature-based explanations with a prediction-only baseline and finds no significant difference in participants’ willingness to switch to the AI’s prediction (p = .66). It concludes that explanations did not perceptibly affect trust, contrary to the claimed increase in over-reliance when the model was wrong. The paper discusses increased acceptance of incorrect predictions in another study, but does not find that effect in its own comparison. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 2.2 Human-AI Decision-Making
What the sources say
- Partly supportedcites [6] 6. Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M Vardoulakis. 2020. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1-12. sourceExperimental — accuracy not yet measured
The “Challenges evaluating AI in clinical settings” section reports that mammography computer-aided detection generally did not improve radiologists’ diagnostic accuracy. That supports an example of human–AI assistance failing to improve performance, but this paper does not establish the broader claim that many empirical studies found no complementary performance, nor does it report a comparative performance result for its own deployment. Page numbers were not available in our copy. (LLM, conf=0.76)
- Supportedcites [16] 16. Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2020. Feature-Based Explanations Don't Help People Detect Misclas- sifications of Online Toxicity. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 95-106. sourceExperimental — accuracy not yet measured
In Accuracy and Agreement, the paper reports that showing participants the model’s prediction slightly reduced accuracy, while explanations did not significantly change that result. Experiment Design for Evaluating Interpretability also describes a gap between hoped-for and demonstrated gains in human–model performance across this and similar studies. The paper therefore supports the claim as one example among the studies cited, rather than as evidence about every human–AI decision task. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [37] 37. Ben Green and Yiling Chen. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-24. source
- Supportedcites [52] 52. Vivian Lai, Han Liu, and Chenhao Tan. 2020. " Why is' Chicago'deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13. sourceExperimental — accuracy not yet measured
The Results (p. 8) show that human participants rarely outperformed the ML model: 0.4% in Experiment 1 and 8.1% with effective real-time assistance in Experiment 2. This study therefore supports the cited sentence as one example of limited complementary performance, while acknowledging that a minority of participants did outperform the model. (LLM, conf=0.88)
- Supportedcites [58] 58. Han Liu, Vivian Lai, and Chenhao Tan. 2021. Understanding the effect of out-of-distribution examples and interactive explanations on human-AI decision making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-45. sourceExperimental — accuracy not yet measured
Section 2.1 reports that, across many studies of challenging prediction tasks, human–AI teams seldom outperform AI alone; it also identifies a notable exception. Because the paper defines complementary performance as outperforming both AI alone and humans alone, those reported studies support the citing sentence’s claim. (LLM, conf=0.91)
- Supportedcites [61] 61. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749-760. source
In the Results section, the paper reports that anaesthesiologists assisted by Prescience performed better than unaided anaesthesiologists but slightly worse than Prescience alone. It therefore supports this citation as one example of a study that did not observe complementary performance; it does not itself establish how many studies found that result. (LLM, conf=0.90)
- Supportedcites [68] 68. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1-52. source
This paper supports its use as one example in the citing sentence’s list of empirical studies. Section 3.2 (page unavailable) reports that, for typical apartments, participants using a model made less accurate predictions than the model alone, although they did better than participants without a model. The paper does not itself establish how many studies found this pattern. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Not checked — no full text of the sourcecites [84] 84. Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. 2020. How do visual explanations foster end users' appropriate trust in machine learning?. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 189-201. source
- Supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In 3.2 Results (page not provided), the paper reports that human–AI joint decisions were not more accurate than AI decisions alone, even in the low-confidence cases where the authors expected a benefit. This study therefore supports the cited sentence as one example of an empirical study that did not observe complementary performance. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our exploratory quantitative analysis shows that, consistent with prior work [4,82,87], featurebased explanations did not improve participants' decision outcomes compared to their performance without AI support, and led to overreliance on incorrect AI predictions.”
- Also stated in: § Abstract · plain
“increased their overreliance on AI”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 1 INTRODUCTION
What the sources say
- Supportedcites [4] 4. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-16. source
Section 3.1 (p. 4) describes the study’s feature-based explanations as highlights showing the importance of input words. The pilot-study findings (Section 3.2, p. 4) report increased reliance on AI recommendations even when they were incorrect; Section 5.1 (p. 8) reports a similar effect for explanations of the top prediction, though the effect varied by task. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [82] 82. Xinru Wang and Ming Yin. 2021. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces. 318-328. source
- Partly supportedcites [87] 87. Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295-305. sourceExperimental — accuracy not yet measured
In 4.2 Results (page unavailable), the paper reports that local feature-based explanations did not improve trust calibration or AI-assisted accuracy. It does not show that they caused greater reliance on incorrect predictions: agreement was similar with and without explanations. Its discussion of increased acceptance of incorrect predictions concerns Lai and Tan’s study, not its own experiment. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“This suggests that example-based explanations can provide additional context and support inductive reasoning [11] to help people form additional intuition about features.”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > Applying intuition about features to reason about explanations.
What the sources say
- Supportedcites [11] 11. Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the 25th international conference on intelligent user interfaces. 454-464. source
Section 3.1 (p. 3) describes example-based explanations that show four additional food images and ask participants to recognize ingredients contributing to fat content and draw their own conclusions. It explicitly identifies this as inductive reasoning, supporting the paraphrase about context and intuition about features. (LLM, conf=0.91)
hedgedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
101 more claims · 1 with their cited sources checked
Create a free account to see all 210 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 52
- It may state two separate claims as one. 63
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings suggest that a reflective approach toward the use of GenAI tools, which can lead to what prior work refers to as”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conversely, with the important caveat that users' self-confidence is a subjective measure of their knowledge, experiences, and abilities on the tasks [20,59,85], higher self-confidence is associated with more critical thinking, even though workers who are confident in their own skills tend to perceive greater effort in these tasks, particularly when evaluating and applying AI responses.”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This can lead to disastrous outcomes through overreliance [19, 25, 83, 102], when people take actions based on incorrect outputs.”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vice versa, people may -and often do -trust an untrustworthy model (Chen, Liao, Vaughan, & Bansal, 2023).”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Chen, Liao, Vaughan, & Bansal, 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They show a number of studies that suggest that explanations can increase users' reliance on AI predictions even if the AI system's predictions are wrong.”
From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 4.3.1 Subjective preferences of explanation types.
“For example, P17 said "when the data was super consistent, I was gonna go with it"”
§ Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations > 4.3.1 Subjective preferences of explanation types.
“"when the data was super consistent, I was gonna go with it" (P32).”
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Explanations can also be classified as global or local [1, 35].”
BackgroundMethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“First, we overview related work on explainable AI (XAI), focusing on XAI for ML models—what is sometimes referred to as interpretability in the ML literature [26, 53].”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 2×
“A diverse set of XAI methods have been proposed that surface technical details of ML models, as surveyed by several authors [1, 2, 18, 35, 38].”
BackgroundExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 2×
“uncertainty awareness [18, 73], we avoid using the term “uncertainty” as participants’ comments reflect their subjective perception of reliability rather than the actual model uncertainty quantity.”
MethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 1×
“, specifically looking for examples with different ground truth labels [53, 82] or using influence functions [50]), we opt for a simple method of selecting the two nearest neighbors from the training set.”
MethodGrounded Theory: An Exploration of Process and Procedure
Walker, D., & Myrick, F. (2006) · Qualitative Health Research · doi:10.1177/1049732305285972 · Cited by 1,146
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991 · cited 1×
“Similar metrics have been used to evaluate the calibration of uncertainty quantification [7, 9, 64]—the estimated uncertainty”
MethodDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 1×
“Additionally, researchers in the AI, HCI, and CSCW communities have called for more human-centered approaches [28, 51, 72, 74] to investigate what people need and how they interact with AI explanations in specific use cases.”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Self-citationDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 9×
“decision tasks with different data types including text data [4, 49, 54], tabular data [61, 73, 78], and image data [10, 45].”
BackgroundResultInfluentialSelf-citationDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Self-citationManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 5×
“decision tasks with different data types including text data [4, 49, 54], tabular data [61, 73, 78], and image data [10, 45].”
BackgroundResultInfluentialGut feelings: the intelligence of the unconscious
(2008) · Choice Reviews Online · doi:10.5860/choice.45-4078 · Cited by 723
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Self-citation'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 1×
“This approach helps people understand not only how the model makes decisions but also how it might make mistakes [8, 25, 49].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 3×
“A diverse set of XAI methods have been proposed that surface technical details of ML models, as surveyed by several authors [1, 2, 18, 35, 38].”
BackgroundBeede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 1×
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundRetiring Adult: New Datasets for Fair Machine LearningPreprint
Ding, F., Hardt, M., Miller, J. A., & Schmidt, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04884 · Cited by 599
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 1×
“The dual processing model has been cited as a useful framework [10, 30, 42, 50, 51]—instead of engaging in analytical reasoning with the explanations (system 2 thinking), people may invoke heuristics or rules-of-thumb to make a quick judgment (system 1 thinking).”
MethodUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“While AI assistance typically provides an ML model’s predictions, there is increasing work studying whether additional information—performance measures [76], explanations [4, 73], or information about prediction uncertainty [62, 78], for example—can further improve decision outcomes [47].”
BackgroundMetrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518
Interpretable Machine Learning: Moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · Queue · doi:10.1145/3511299 · Cited by 516
Self-citationExpertise-Based Intuition and Decision Making in Organizations
Salas, E., Rosen, M. A., & DiazGranados, D. (2010) · Journal of Management · doi:10.1177/0149206309350084 · Cited by 509 · cited 2×
“Intuition has therefore been widely studied for expert decision-making [72] and decision-making under uncertainty [40].”
BackgroundHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 480
Self-citationBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“We can also anticipate that people whose own intuition outcomes are highly consistent with the AI system will benefit less from AI support in general because of a weak pathway 1, which has been alluded to by prior works on complementary knowledge [3, 78].”
BackgroundSelf-citationThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“1 Can XAI methods improve appropriate reliance? Prior work has asked whether providing decision-makers with explanations of an AI system’s predictions can improve appropriate reliance, since decision-makers might be less likely to follow an AI prediction if an explanation suggests flawed model reasoning [5, 12, 17].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 1×
“Different from qualitative studies that aim to understand the experience of decision-makers retrospectively [14, 45, 47, 65, 80], we adopt a think-aloud protocol to investigate the real-time process as participants engage with the AI output [43, 86], and use the data to understand why two types of explanations—feature…”
MethodExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M., & Krishna, R. (2022) · arXiv · Cited by 403
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
The Elaboration Likelihood Model
Petty, R. E., & Briñol, P. (2012) · Handbook of Theories of Social Psychology: Volume 1 · doi:10.4135/9781446249215.n12 · Cited by 324
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Reviewing intuitive decision‐making and uncertainty: the implications for medical education
Hall, K. H. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01140.x · Cited by 302
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300 · cited 1×
“A similar metric has been used to evaluate the calibration of uncertainty quantification [7]—the reported uncertainty should be highly correlated with the observed error rate.”
MethodSelf-citationExplaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
Self-citationDisparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274 · cited 1×
“Additionally, researchers in the AI, HCI, and CSCW communities have called for more human-centered approaches [29, 55, 81, 83] to investigate what people need and how they interact with AI explanations in specific use cases.”
BackgroundEvaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254 · cited 1×
“One is to select prototypes that are representative of a prediction class to help people understand why the model predicts the current instance belongs to that class [11, 13, 41, 49].”
BackgroundA Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 1×
“Human-AI decision-making, also referred to as AI-assisted decision-making, broadly encompasses set-ups where an ML model is used to help users to make a final judgment or decision [22, 34, 47]—a form of collaboration between human and AI system.”
MethodThe effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220 · cited 2×
“Wang and Yin [73] found that example-based explanations underperform feature-based explanations for improving appropriate reliance, though other studies suggest example-based explanations are better at helping people detect model errors [10, 13, 45], usually by recognizing the dissimilarity between the instance and se…”
BackgroundAutomated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“Different from a small but growing number of qualitative studies that aim to understand the experience of decision-makers retrospectively [14, 41, 43, 60, 71], we adopt a think-aloud protocol to investigate the real-time process as participants engage with AI output [39, 77], and use the data to understand why two typ…”
MethodPost-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207 · cited 1×
“For the biography classification task, we used a standard approach [58] of color-highlighting the words (features), particularly those with the top 10 highest-magnitude coefficients, as shown in Figure 1 (c).”
MethodInterpretable machine learning: moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2022) · Communications of the ACM · doi:10.1145/3546036 · Cited by 203
Self-citationHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 1×
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundIntuition: A Review of the Literature
Shirley, D. A., & Langan-Fox, J. (1996) · Psychological Reports · doi:10.2466/pr0.1996.79.2.563 · Cited by 193
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · cited 1×
“, weight), which align with prior findings about how experts critique model explanations in interactive ML settings [33, 75].”
BackgroundJacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177 · cited 1×
“Different from a small but growing number of qualitative studies that aim to understand the experience of decision-makers retrospectively [14, 41, 43, 60, 71], we adopt a think-aloud protocol to investigate the real-time process as participants engage with AI output [39, 77], and use the data to understand why two typ…”
MethodDesigning for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
Self-citation'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
HIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V., Fong, R. C., & Russakovsky, O. (2021) · European Conference on Computer Vision · doi:10.48550/arxiv.2112.03184 · Cited by 153
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 1×
“Different from a small but growing number of qualitative studies that aim to understand the experience of decision-makers retrospectively [14, 41, 43, 60, 71], we adopt a think-aloud protocol to investigate the real-time process as participants engage with AI output [39, 77], and use the data to understand why two typ…”
MethodInkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · cited 1×
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
Self-citationBayati, M., Braverman, M., Gillam, M., Mack, K. M., Ruiz, G., Smith, M. S., & Horvitz, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109264 · Cited by 122 · cited 1×
“Providing decision-makers with explanations of the AI’s predictions is thought to improve appropriate reliance because decision-makers will be less likely to follow the AI prediction if an explanation suggests flawed model reasoning [5, 12, 17].”
BackgroundHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
Self-citationSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
Park, H., Ahn, D., Hosanagar, K., & Lee, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445304 · Cited by 116
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111 · cited 7×
“previous studies that used the same tasks [53, 76], this experience accuracy also ensures that the AI and human-alone perform at a comparable level.”
BackgroundResultInfluentialKnowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
Liao, Q., Zhang, Y., Luss, R., et al. (2022) · AAAI Conference on Human Computation & Crowdsourcing · doi:10.48550/arxiv.2206.10847 · Cited by 107
Self-citationExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
Self-citationSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
Rechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97 · cited 1×
“While AI assistance typically provides an ML model’s predictions, there is growing work studying whether additional information—performance measures [85], explanations [4, 82], or information about prediction uncertainty [69, 87], for example—can further improve decision outcomes [51].”
BackgroundTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · cited 7×
“Prior works that involved controlled experiments on human-AI decision-making have typically adopted one of two workflows [51]: The participant either saw a prediction from an AI system and then made their own decision, or the participant made their own decision first and then was given an opportunity to update their d…”
BackgroundMethodInfluentialSelf-citationHIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72
Understanding User Reliance on AI in Assisted Decision-Making
Cao, S., & Huang, C.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555572 · Cited by 71
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
Self-citationTowards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58 · cited 1×
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundMachine Explanations and Human Understanding
Chen, C., Feng, S., Sharma, A., & Tan, C. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593970 · Cited by 53
Improving understandability of feature contributions in model-agnostic explainable AI tools
Hadash, S., Willemsen, M. C., Snijders, C. C. P., & IJsselsteijn, W. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517650 · Cited by 51
A Human-Centered Agenda for Intelligible Machine Learning
Vaughan, J. W., & Wallach, H. (2021) · Machines We Trust · doi:10.7551/mitpress/12186.003.0014 · Cited by 50
Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining
Erlei, A., Nekdem, F., Meub, L., Anand, A., & Gadiraju, U. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7462 · Cited by 48 · cited 1×
“Human-AI decision-making, also referred to as AI-assisted decision-making, broadly encompasses set-ups where an ML model is used to help users to make a final judgment or decision [23, 31, 36, 51]—often considered as a form of collaboration between human and AI system.”
MethodRosenblatt, A. D., & Thickstun, J. T. (1994) · The Psychoanalytic Quarterly · doi:10.1080/21674086.1994.11927433 · Cited by 37 · cited 2×
“While various definitions of “intuition” exist [34, 71, 74], we follow recent work on human-AI decision-making [19, 67] and use the term to broadly refer to beliefs or heuristics that people bring into the decision based on their domain knowledge, experience, instinct, or pattern recognition.”
BackgroundHeterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening
Cheng, L., & Chouldechova, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555101 · Cited by 31
Heterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening
Cheng, L., & Chouldechova, A. (2022) · arXiv · doi:10.48550/arxiv.2204.05478 · Cited by 20
Machines We Trust: Perspectives on Dependable Ai
Pelillo, M., & Scantamburlo, T. (2021) · Cited by 18
How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?Preprint
Yang, F. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/s9q8t · Cited by 18
Role of Data and Intuition in Decision Making Processes
Potančok, M. (2019) · doi:10.20470/jsi.v10i3.377 · Cited by 10
(2017) · doi:10.9728/dcs.2024.25.7.1847 · Cited by 2
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.13509
Self-citationVerification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“Similar metrics have been used to evaluate the calibration of uncertainty quantification [7, 9, 64]—the estimated uncertainty”
MethodHuman-centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · arXiv · doi:10.48550/arxiv.2002.01092
Research works printed but not linked: 1
- Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning. PMLR, 2668--2677.
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
“Human-AI decision-making (collaboration): In recent autonomous vehicle systems, machine learning models are utilized to assist users in making final judgments or decisions, representing a form of collaboration between humans and AI systems [183].”
MethodA literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“The data was obtained using queries that include keywords such as ‘Explainable AI,’ ’XAI,’ and related terms through Google Scholar’s advanced search [111], [120].”
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Self-citation
Metadata
- Authors
- Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G.
- Affiliations
- Carnegie Mellon University (US)
- Microsoft Research Montréal (Canada) (CA)
- Microsoft Research, New York, NY, USA
- Microsoft (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-28
- Volume
- 7
- Issue
- CSCW2
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4387344906
- Cited by
- 220
- References
- 74
- Citations by year
- 2026: 90 · 2025: 72 · 2024: 47 · 2023: 10
- FWCI
- 15.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Institut National Du Cancer
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human intuition, AI explanations, human-AI decision-making, overreliance on AI, feature-based explanations, example-based explanations, human-AI complementarity, decision support system, think-aloud
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.