Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index; 4 of the 10 sources its claims cite were checked for support.

Dynamic Selection of Reliance Calibration Cues With AI Reliance Model

Open access
Fukuchi, Yosuke, Yamada, Seiji · 2023 · IEEE Access

Abstract

Understanding what an AI system can and cannot do is necessary for end-users to use the AI properly without being over- or under-reliant on it. Reliance calibration cues (RCCs) communicate an AI’s capability to users, resulting in optimizing their reliance on it. Previous studies have typically focused on continuously presenting RCCs, and although providing an excessive amount of RCCs is sometimes problematic, limited consideration has been given to the question of how an AI can selectively provide RCCs. This paper proposes vPred-RC, an algorithm in which an AI decides whether to provide an RCC and which RCC to provide. It evaluates the influence of an RCC on user reliance with a cognitive model that predicts whether a human will assign a task to an AI agent with or without an RCC. We tested vPred-RC in a human-AI collaborative task called the collaborative CAPTCHA (CC) task. First, our reliance prediction model was trained on a dataset of human task assignments for the CC task and found to achieve 83.5% accuracy. We further evaluated vPred-RC’s dynamic RCC selection in a user study. As a result, the RCCs selected by vPred-RC enabled participants to more accurately assign tasks to an AI when and only when the AI succeeded compared with randomly selected ones, suggesting that vPred-RC can successfully calibrate human reliance with a reduced number of RCCs. The selective presentation of RCCs has the potential to enhance the efficiency of collaboration between humans and AIs with fewer communication costs.

Overview

CoChat Summary

TL;DR

This paper proposes vPred-RC, which selectively decides whether to show reliance calibration cues (RCCs) and which cue to show by predicting how a cue will affect human reliance. In a collaborative CAPTCHA task, the reliance model reached 83.5% accuracy, and vPred-RC improved task-assignment accuracy relative to random cue selection while using fewer RCCs.

Key findings

  • The reliance prediction model trained on the collaboration-history dataset achieved a maximum accuracy of 83.5% (SD=1.1%) on average at the 18th epoch with three-fold cross validation.
  • In the vPred-RC evaluation, ANCOVA showed significant effects of the number of cues (F(1,110)=7.629, p=.0067, ηp²=.065) and the RCC selection method (F(1,110)=6.575, p=.012, ηp²=.056) on the F-score of human decisions.
  • In the reliance-dataset analysis, the threshold strategy had slightly higher scores than the proportional strategy, but the difference was not significant (F(1,141)=1.647, p=.20, ηp²=.012).
  • For subjective measures, participants rated RCCs selected by vPred-RC as more useful for assignment (F=3.9477, p=.0494) and for understanding AI performance (F=4.2278, p=.0421); “AI trustable” was also significant in favor of vPred-RC (F=6.0794, p=.0152).

Methods

  • The paper introduces vPred-RC, a method that compares predicted reliance with and without an RCC, then selects the cue that best reduces the discrepancy between human reliance rate and AI success probability, subject to a threshold θ.
  • Data came from a collaborative CAPTCHA (CC) task: 228 crowdworkers were recruited for reliance-data collection, 145 completed that task; for evaluation, 170 were recruited and 98 completed the CC task.
  • The reliance model used a Transformer encoder-based architecture trained on collaboration history; the task AI for CAPTCHA recognition used ResNet-18, and evaluation used ANCOVA on F-scores and subjective ratings.

Limitations

  • The paper states that vPred-RC is not immediately applicable to tasks where dynamics during each trial matter, such as autonomous driving, and that it assumes task results can be categorized only as success or failure.
  • The text makes evident that the experiments are limited to a CAPTCHA-based crowdworker setting with mostly Japanese participants; it also notes a post-hoc power analysis of 0.73 and that the reliance model still had failures due to poor prediction, indicating room for improvement.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2023.3339548

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 2 Not valid · 6 Not checked · 80 Own results and interpretations

26 more claims

Create a free account to see all 92 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 46
  • It may state two separate claims as one. 36

Create a free account to see these claims.

Citations

Cites

Show all 35 references

Cited by

See all 16

Metadata

Authors
Fukuchi, Y., & Yamada, S.
Affiliations
National Institute of Informatics (JP)
Venue
IEEE Access
Published
2023
Volume
11
Pages
138870-138881
ISSN
2169-3536 (ISSN-L)
OpenAlex
W4389352627
Cited by
16
References
35
Citations by year
2026: 5 · 2025: 6 · 2024: 4 · 2023: 1
FWCI
2.91
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
Japan Science and Technology Agency (JST) Core Research for Evolutional Science and Technology (CREST), Japan (JPMJCR21D4)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
AI reliance, human-AI collaboration, cognitive models, human-AI communication, overreliance on AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Dynamic Selection of Reliance Calibration Cues With AI Reliance Model. https://app.cochat.ai/papers/10.1109/access.2023.3339548
The paper
Fukuchi, Y., & Yamada, S. (2023). Dynamic Selection of Reliance Calibration Cues With AI Reliance Model. IEEE Access. https://doi.org/10.1109/access.2023.3339548

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.