Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 23 of the 42 sources its claims cite were checked for support.

Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

Open access
Zhang, Yunfeng, Liao, Q. Vera, Bellamy, Rachel K. E. · 2020 · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

Abstract

Today, AI is being increasingly used to help human experts make decisions in high-stakes scenarios. In these scenarios, full automation is often undesirable, not only due to the significance of the outcome, but also because human experts can draw on their domain knowledge complementary to the model's to ensure task success. We refer to these scenarios as AI-assisted decision making, where the individual strengths of the human and the AI come together to optimize the joint decision outcome. A key to their success is to appropriately calibrate human trust in the AI on a case-by-case basis; knowing when to trust or distrust the AI allows the human expert to appropriately apply their knowledge, improving decision outcomes in cases where the model is likely to perform poorly. This research conducts a case study of AI-assisted decision making in which humans and AI have comparable performance alone, and explores whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. Specifically, we study the effect of showing confidence score and local explanation for a particular prediction. Through two human experiments, we show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making, which may also depend on whether the human can bring in enough unique knowledge to complement the AI's errors. We also highlight the problems in using local explanation for AI-assisted decision making scenarios and invite the research community to explore new approaches to explainability for calibrating human trust in AI.

Overview

CoChat Summary

## TL;DR

This paper studies whether showing case-specific AI confidence scores and local explanations helps people calibrate trust in AI-assisted income prediction. Confidence scores improved trust calibration, but neither confidence nor explanations improved the joint decision accuracy.

## Key findings

  • In Experiment 1, showing AI confidence scores significantly improved trust calibration: the main effect of showing confidence on switch percentage was significant, F(1,64) = 4.64, p = .035, and the interaction with confidence level was significant, F(4,256) = 15.8, p < .001.
  • Trust calibration held whether or not the AI prediction was shown: the three-way interaction between confidence, prediction, and confidence level was not significant, F(4,256) = 0.266, p = .899.
  • Showing confidence did not improve AI-assisted accuracy: there was no significant difference across prediction and confidence conditions, and H2 was rejected.
  • In Experiment 2, local explanations did not improve trust calibration or accuracy: switch percentage showed no significant difference between explanation and baseline (p = .66), and AI-assisted accuracy also did not differ significantly across conditions, F(2,24) = 0.810, p = .457.

## Methods

  • Two human-subject experiments on an AI-assisted binary income prediction task using the UCI Adult dataset (48,842 instances, 14 attributes); participants predicted whether annual income exceeded $50K.
  • Experiment 1 recruited 72 Amazon Mechanical Turk participants and used a 2×2×2 design: show vs. not show AI confidence, show vs. not show AI prediction, and full vs. partial model, with 8 conditions and 9 participants per condition.
  • Experiment 2 recruited 9 participants and used local explanations generated by the Shapley method; analysis relied on behavioral trust measures (switch percentage, agreement percentage) and ANOVA.

## Limitations

  • The paper states that participants were not experts in income prediction; this was mitigated with training and domain statistics, but experts would be more realistic and were not used.
  • The task was contrived and participants were not actually responsible for real outcomes; the authors partially mitigated this with bonus incentives, but note that more realistic high-stakes studies would need caution.
  • The method for calibrating trust via confidence depends on model probabilities being well calibrated; the paper notes this may not hold for some models such as SVM, though Platt scaling or isotonic regression could address it.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3351095.3372852

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

21 Valid · 3 Not valid · 6 Not checked · 64 Own results and interpretations

46 more claims · 3 with their cited sources checked

Create a free account to see all 94 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 28
  • It may state two separate claims as one. 26
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

53 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 25 references

Cited by

See all 895

Metadata

Authors
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E.
Affiliations
IBM (United States) (US)
Venue
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2020-01-27
Pages
295-305
OpenAlex
W2999637955
Cited by
895
References
25
Citations by year
2026: 229 · 2025: 163 · 2024: 198 · 2023: 128 · 2022: 89 · 2021: 71 · 2020: 17
FWCI
29.12
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
AI-assisted decision-making, trust calibration, confidence scores, local explanations, explainability, human-AI collaboration, human expertise, trust in AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. https://app.cochat.ai/papers/10.1145/3351095.3372852
The paper
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3351095.3372852

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.