The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 23 of the 42 sources its claims cite were checked for support.
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making
Open accessAbstract
Today, AI is being increasingly used to help human experts make decisions in high-stakes scenarios. In these scenarios, full automation is often undesirable, not only due to the significance of the outcome, but also because human experts can draw on their domain knowledge complementary to the model's to ensure task success. We refer to these scenarios as AI-assisted decision making, where the individual strengths of the human and the AI come together to optimize the joint decision outcome. A key to their success is to appropriately calibrate human trust in the AI on a case-by-case basis; knowing when to trust or distrust the AI allows the human expert to appropriately apply their knowledge, improving decision outcomes in cases where the model is likely to perform poorly. This research conducts a case study of AI-assisted decision making in which humans and AI have comparable performance alone, and explores whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. Specifically, we study the effect of showing confidence score and local explanation for a particular prediction. Through two human experiments, we show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making, which may also depend on whether the human can bring in enough unique knowledge to complement the AI's errors. We also highlight the problems in using local explanation for AI-assisted decision making scenarios and invite the research community to explore new approaches to explainability for calibrating human trust in AI.
Overview
CoChat Summary
## TL;DR
This paper studies whether showing case-specific AI confidence scores and local explanations helps people calibrate trust in AI-assisted income prediction. Confidence scores improved trust calibration, but neither confidence nor explanations improved the joint decision accuracy.
## Key findings
- In Experiment 1, showing AI confidence scores significantly improved trust calibration: the main effect of showing confidence on switch percentage was significant, F(1,64) = 4.64, p = .035, and the interaction with confidence level was significant, F(4,256) = 15.8, p < .001.
- Trust calibration held whether or not the AI prediction was shown: the three-way interaction between confidence, prediction, and confidence level was not significant, F(4,256) = 0.266, p = .899.
- Showing confidence did not improve AI-assisted accuracy: there was no significant difference across prediction and confidence conditions, and H2 was rejected.
- In Experiment 2, local explanations did not improve trust calibration or accuracy: switch percentage showed no significant difference between explanation and baseline (p = .66), and AI-assisted accuracy also did not differ significantly across conditions, F(2,24) = 0.810, p = .457.
## Methods
- Two human-subject experiments on an AI-assisted binary income prediction task using the UCI Adult dataset (48,842 instances, 14 attributes); participants predicted whether annual income exceeded $50K.
- Experiment 1 recruited 72 Amazon Mechanical Turk participants and used a 2×2×2 design: show vs. not show AI confidence, show vs. not show AI prediction, and full vs. partial model, with 8 conditions and 9 participants per condition.
- Experiment 2 recruited 9 participants and used local explanations generated by the Shapley method; analysis relied on behavioral trust measures (switch percentage, agreement percentage) and ANOVA.
## Limitations
- The paper states that participants were not experts in income prediction; this was mitigated with training and domain statistics, but experts would be more realistic and were not used.
- The task was contrived and participants were not actually responsible for real outcomes; the authors partially mitigated this with bonus incentives, but note that more realistic high-stakes studies would need caution.
- The method for calibrating trust via confidence depends on model probabilities being well calibrated; the paper notes this may not hold for some models such as SVM, though Platt scaling or isotonic regression could address it.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
21 Valid · 3 Not valid · 6 Not checked · 64 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, the decision performance was linked to monetary bonus, with a reward of 5 cents if the final prediction was correct and a loss of 2 cents if otherwise (in addition to a base pay of $3). Prior research showed that such a reward design is effective in motivating participants to optimize the decision outcome [2, 31]”
What the sources say
- Supportedcites [2] [2] Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, and Eric Horvitz. 2019. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In AAAI Conference on Artificial Intelligence . AAAI. https://www.microsoft.com/enus/research/publication/updates-in-human-ai-teams-understanding-andaddressing-the-performance-compatibility-tradeoff/ - source
The Experiments section (pp. 4–5) describes monetary rewards and penalties tied to workers’ decisions. It says the payoff design was a good incentive to learn when to trust the AI and that doing so was the only way to earn a higher score. The citing paper’s specific 5-cent reward and 2-cent loss are its own design, not amounts attributed to this source. (LLM, conf=0.88)
- Partly supportedcites [31] [31] Forrest W Young. 1967. Twelve-choice probability learning with payoffs. Psychonomic Science 7, 10 (1967), 353-354. - sourceExperimental — accuracy not yet measured
The introduction (p. 1) reports that monetary payoffs led participants in earlier experiments to more nearly maximize earnings, and the Results and Discussion (p. 2) show that choices in this study strongly reflected expected values. But this study did not test the cited reward design: its methods (p. 2) state that incorrect predictions incurred no loss, whereas the citing sentence describes a two-cent loss. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A key to success in AI-assisted decision making is to form a correct mental model of the model's error boundaries [2].”
What the sources say
- Supportedcites [2] [2] Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, and Eric Horvitz. 2019. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In AAAI Conference on Artificial Intelligence . AAAI. https://www.microsoft.com/enus/research/publication/updates-in-human-ai-teams-understanding-andaddressing-the-performance-compatibility-tradeoff/ - source
The Introduction (p. 1) says that successful AI-assisted decisions depend on knowing when to follow or override the AI, which generally requires a mental model of when to trust it. The Q1 section (p. 5) reports that workers whose mental models matched the AI’s true error boundary had the highest team performance. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In contrast, prior works studying AI-assisted decision-making often used setups where humans' decision performance was significantly inferior than the model's [17, 30], which would by default reward people to rely on the AI.”
What the sources say
- Supportedcites [17] [17] Vivian Lai and Chenhao Tan. 2019. On Human Predictions with Explanations and Predictions of Machine Learning Models. In Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* '19 . ACM Press, New York, New York, USA, 29-38. https://doi.org/10.1145/3287560.3287590 - source
This paper is an example of the setup described. The Experimental Setup (p. 3) reports a model accuracy of 87% and says machine performance surpassed human performance of about 50% by a wide margin; the Concluding Discussion (p. 9) makes the same comparison. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [30] [30] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Understanding the effect of accuracy on trust in machine learning models. In Conference on Human Factors in Computing Systems - Proceedings . Association for Computing Machinery. https://doi.org/10.1145/3290605.3300509 - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“several recent empirical studies found little evidence that the level of transparency has significant impact on people's willingness to trust a ML system, whether by using a directly interpretable model, allowing user to inspect the model behavior, showing explanation or reducing the number of features presented [5, 16, 27, 29].”
What the sources say
- Not checked — no full text of the sourcecites [5] [5] Hao Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O'Connell, Terrance Gray, F. Maxwell Harper, and Haiyi Zhu. 2019. Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In Conference on Human Factors in Computing Systems - Proceedings . Association for Computing Machinery. https://doi.org/10.1145/3290605.3300789 - source
- Not checked — no full text of the sourcecites [16] [16] Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . ACM, 487. - source
- Partly supportedcites [27] [27] Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and Measuring Model Interpretability. (feb 2018). arXiv:1802.07810 https://arxiv.org/pdf/1802.07810. pdfhttp://arxiv.org/abs/1802.07810 - sourceExperimental — accuracy not yet measured
The Discussion and Conclusion (page unavailable) reports no significant improvement in how closely participants followed a clear, few-feature model’s predictions compared with a black-box, more-feature model. That supports the claim about reliance on predictions, but not an unqualified claim about willingness to trust: section 3.2 reports significantly greater stated confidence in predictions from the clear, two-feature model. This paper tested model transparency and feature count, not all the approaches listed in the citing sentence. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“frequently updating the AI algorithm may cause confusion to the human decision-maker, who may accept or reject the AI's recommendations at a wrong time, even if the algorithm's overall performance improved [2].”
What the sources say
- Supportedcites [2] [2] Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, and Eric Horvitz. 2019. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In AAAI Conference on Artificial Intelligence . AAAI. https://www.microsoft.com/enus/research/publication/updates-in-human-ai-teams-understanding-andaddressing-the-performance-compatibility-tradeoff/ - source
The Introduction (p. 1) says team success depends on the human knowing when to follow or override AI recommendations, and that updates can disrupt those expectations even when AI performance improves. The patient-readmission example (p. 2) shows a more accurate updated classifier leading a doctor with an outdated mental model to take wrong actions. The Discussion (p. 8) also notes that developers may push updates frequently. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
46 more claims · 3 with their cited sources checked
Create a free account to see all 94 claims and every source verdict.
- Probably not a substantial claim. 28
- It may state two separate claims as one. 26
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“On the contrary, several studies found that showing feature-based explanations increases people's tendency to over-rely on the model when it is wrong, compared to showing only the AI predictions [4,68,82,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [87]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“since the mere presence of an explanation increases AI trust (Zhang, Liao, and Bellamy 2020).”
From Human-Clinical AI Agent Collaboration (2025) · cites this paper as Zhang, Liao, and Bellamy 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“This is achieved by, for example, presenting information about a system's confdence throughout the interaction [90, 104, 145] using cues or warning signals or by increasing a system's transparency through explanations [108].”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [145]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Potential cognitive mechanisms that underlie these results suggest that the mere presence of an explanation increases trust [96,97], and that explanations anchor humans to the prediction [9,91].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [97]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that communicating explanations, on average, is not improving trust calibration, i.e., users still end-up in situations where they over-trust or under-trust AI-based recommendation.”
From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [96]
Experimental — accuracy not yet measured
53 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Developing and validating trust measures for e-commerce: an integrative typology
McKnight, D. H., Choudhury, V., & Kacmar, C. (2002) · Information Systems Research · doi:10.1287/isre.13.3.334.81 · Cited by 5,550 · cited 1×
“The concept of trust has its roots in relationships between humans, reflected in many aspects of collaborative behaviors with others such as willingness to depend, give information and make purchase [12, 21].”
BackgroundExplanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“Income category [83] Accuracy ↑ 65% 75% 73% ✗”
BackgroundMethodVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
Self-citationTrust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16553 · Cited by 469
Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021
Chaudhry, M. A., & Kazim, E. (2022) · AI and Ethics · doi:10.1007/s43681-021-00074-z · Cited by 425
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
“In responding to this call, there is recently a growing line of literature on empirically evaluating the effectiveness of XAI methods (e.g., [14, 16, 41, 69, 71]).”
BackgroundMethodInfluentialTrusting Automation: Designing for Responsivity and Resilience
Chiou, E. K., & Lee, J. D. (2023) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208211009995 · Cited by 301
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
“In most cases, the combination (also known as “AI-assisted decision making”) will obtain the best results [24].”
BackgroundChoi, S., Jang, Y., & Kim, H. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2049145 · Cited by 294
Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024) · JMIR AI · doi:10.2196/53207 · Cited by 305
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Chong, L., Zhang, G., Goucher-Lambert, K., Kotovsky, K., & Cagan, J. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107018 · Cited by 262
How do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 281
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300
Self-citationThree Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216
Self-citationGomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Leichtmann, B., Humer, C., Hinterreiter, A., Streit, M., & Mara, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107539 · Cited by 200
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208
“Recently, a new line of research has emerged that argues that maximising AI reliance does not fully exploit the potential of state-of-the-art human-AI decision making [6, 9, 62, 78].”
BackgroundExploring automation bias in human–AI collaboration: a review and implications for explainable AI
Romeo, G., & Conti, D. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02422-7 · Cited by 209
“Merely making AI explainable or transparent is not enough: explanations must actually help users recognize uncertainty or errors (Zhang et al. 2020).”
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170
Fairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161
Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
Self-citationHow Explainability Contributes to Trust in AI
Ferrario, A., & Loi, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533202 · Cited by 163
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
Self-citationMa, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Llorca, D. F., Yela, D. F., Junklewitz, H., Scalzo, S., Mazzini, G., Sánchez, I., Garrido, J. S., et al. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594069 · Cited by 270
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 254
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Self-citationHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
Self-citationWischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“inform users about the specifc capabilities and limitations of a system, such as information about a system’s confdence [90, 104, 145] or general information about a system’s reliability [61, 69].”
BackgroundInfluentialZhang, R., Duan, W., Flathmann, C., McNeese, N., Freeman, G., & Williams, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610072 · Cited by 134
Bayesian modeling of human–AI complementarity
Mark Steyvers, B., Tejeda, H., Kerrigan, G., & Smyth, P. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2111547119 · Cited by 249
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
Graziani, M., Dutkiewicz, L., Calvaresi, D., Amorim, J. P., Yordanova, K., Vered, M., Nair, R. S., Abreu, P. H., Blanke, T., Pulignano, V., et al. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10256-8 · Cited by 191
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502104 · Cited by 214
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 338
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Thus, numerous studies have highlighted the importance of calibrated trust in improving AI-assisted decision-making [7, 9, 31, 44, 51, 53, 67].”
BackgroundSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
Nourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
Drozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
Complementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 236
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118
Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., Riener, A., & Riedl, M. O. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503727 · Cited by 117
Self-citationDikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
Self-citationDoes Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106
Effect of AI Explanations on Human Perceptions of Patient-Facing AI-Powered Healthcare Systems
Zhang, Z., Genc, Y., Wang, D., Ahsen, M. E., & Fan, X. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01743-6 · Cited by 113
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
Operationalizing Human-Centered Perspectives in Explainable AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Mara, M., Streit, M., Wachter, S., Riener, A., & Riedl, M. O. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3441342 · Cited by 123
Self-citationKnowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
“Topics like performance feedback [2, 55], risk perception [27, 34], uncertainty [77] and confdence [10, 79, 87] of machine learning models, impact of explanations [3, 46] have been extensively studied in human-AI decision making.”
BackgroundInfluentialShamszare, H., & Choudhury, A. (2023) · Healthcare · doi:10.3390/healthcare11162308 · Cited by 158
Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2101698 · Cited by 104
Zhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642343 · Cited by 99
“In the rest of this section, we summarize participants’ feedback on each function of our system.”
BackgroundResultAnik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103
“Who goes first? Influences of human-AI workflow on decision making in clinical imaging,”
Fogliato, R., Chappidi, S., Lungren, M., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K., & Nushi, B. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533193 · Cited by 103
Rechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
Hemmer, P., Westphal, M., Schemmer, M., Vetter, S., Vössing, M., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584052 · Cited by 93
On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659051 · Cited by 91
A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Nitsche, M., Kühl, N., & Vössing, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534128 · Cited by 91
Okolo, C. T., Kamath, S., Dell, N. L., & Vashistha, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445420 · Cited by 123
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
“During the 60-minute retrospective interview sessions, we asked participants about their general experience with AI tools and then asked them to walk through the significant moments they collected during the prior week.”
BackgroundMethodInfluentialYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., Choudhury, M. D., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 149
How Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions
Cheng, H.-F., Stapleton, L., Kawakami, A., Sivaraman, V., Cheng, Y., Qing, D., Perer, A., Holstein, K., Wu, Z. S., & Zhu, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501831 · Cited by 134
Improving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
Navigating the Power of Artificial Intelligence in Risk Management: A Comparative Analysis
Yazdi, M., Zarei, E., Adumene, S., & Beheshti, A. (2024) · Safety · doi:10.3390/safety10020042 · Cited by 116
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
“Although prior work [4, 51, 108] has prominently investigated confidence scores in classification tasks with subjective and objec-tive metrics, few studies have focused on how people perceive, interpret and make use of uncertainty (e.g., to understand the role of uncertainty communication in…”
BackgroundMethodInfluentialSelf-citationFok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Khojah, R., Mohamad, M., Leitner, P., & Neto, F. G. D. O. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660788 · Cited by 165
Human-AI Collaboration in a Cooperative Game Setting: Measuring Social Perception and Outcomes
Ashktorab, Z., Liao, Q., Dugan, C., Johnson, J. M., Pan, Q., Zhang, W., Kumaravel, S., & Campbell, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415167 · Cited by 119
Self-citationHow Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)
Schlicker, N., Baum, K., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2022) · doi:10.31234/osf.io/qhwvx · Cited by 79
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 74
XAIR: A Framework of Explainable AI in Augmented Reality
Xu, X., Yu, A., Jonker, T. R., Todi, K., Lu, F., Qian, X., Belo, J. M. E., Wang, T., Li, M., Mun, A., et al. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581500 · Cited by 108
HIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72
Levy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73
“They’ve investigated factors including reported model confidence, model explanations, overall model accuracy, and initial model accuracy [34, 37, 53, 54].”
BackgroundExplainable recommendation: when design meets trust calibration
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2021) · World Wide Web · doi:10.1007/s11280-021-00916-0 · Cited by 68
“These findings provide a possible interpretation for why exposure to explanations did not improve trust calibration [16, 96].”
BackgroundInfluential
Metadata
- Authors
- Zhang, Y., Liao, Q. V., & Bellamy, R. K. E.
- Affiliations
- IBM (United States) (US)
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 295-305
- OpenAlex
- W2999637955
- Semantic Scholar
- 5cc4100a67fd6f2ce3c760655ba7a12f358c7950
- Cited by
- 895
- References
- 25
- Citations by year
- 2026: 229 · 2025: 163 · 2024: 198 · 2023: 128 · 2022: 89 · 2021: 71 · 2020: 17
- FWCI
- 29.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- AI-assisted decision-making, trust calibration, confidence scores, local explanations, explainability, human-AI collaboration, human expertise, trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.