The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index; 4 of the 10 sources its claims cite were checked for support.
Dynamic Selection of Reliance Calibration Cues With AI Reliance Model
Open accessAbstract
Understanding what an AI system can and cannot do is necessary for end-users to use the AI properly without being over- or under-reliant on it. Reliance calibration cues (RCCs) communicate an AI’s capability to users, resulting in optimizing their reliance on it. Previous studies have typically focused on continuously presenting RCCs, and although providing an excessive amount of RCCs is sometimes problematic, limited consideration has been given to the question of how an AI can selectively provide RCCs. This paper proposes vPred-RC, an algorithm in which an AI decides whether to provide an RCC and which RCC to provide. It evaluates the influence of an RCC on user reliance with a cognitive model that predicts whether a human will assign a task to an AI agent with or without an RCC. We tested vPred-RC in a human-AI collaborative task called the collaborative CAPTCHA (CC) task. First, our reliance prediction model was trained on a dataset of human task assignments for the CC task and found to achieve 83.5% accuracy. We further evaluated vPred-RC’s dynamic RCC selection in a user study. As a result, the RCCs selected by vPred-RC enabled participants to more accurately assign tasks to an AI when and only when the AI succeeded compared with randomly selected ones, suggesting that vPred-RC can successfully calibrate human reliance with a reduced number of RCCs. The selective presentation of RCCs has the potential to enhance the efficiency of collaboration between humans and AIs with fewer communication costs.
Overview
CoChat Summary
TL;DR
This paper proposes vPred-RC, which selectively decides whether to show reliance calibration cues (RCCs) and which cue to show by predicting how a cue will affect human reliance. In a collaborative CAPTCHA task, the reliance model reached 83.5% accuracy, and vPred-RC improved task-assignment accuracy relative to random cue selection while using fewer RCCs.
Key findings
- The reliance prediction model trained on the collaboration-history dataset achieved a maximum accuracy of 83.5% (SD=1.1%) on average at the 18th epoch with three-fold cross validation.
- In the vPred-RC evaluation, ANCOVA showed significant effects of the number of cues (F(1,110)=7.629, p=.0067, ηp²=.065) and the RCC selection method (F(1,110)=6.575, p=.012, ηp²=.056) on the F-score of human decisions.
- In the reliance-dataset analysis, the threshold strategy had slightly higher scores than the proportional strategy, but the difference was not significant (F(1,141)=1.647, p=.20, ηp²=.012).
- For subjective measures, participants rated RCCs selected by vPred-RC as more useful for assignment (F=3.9477, p=.0494) and for understanding AI performance (F=4.2278, p=.0421); “AI trustable” was also significant in favor of vPred-RC (F=6.0794, p=.0152).
Methods
- The paper introduces vPred-RC, a method that compares predicted reliance with and without an RCC, then selects the cue that best reduces the discrepancy between human reliance rate and AI success probability, subject to a threshold θ.
- Data came from a collaborative CAPTCHA (CC) task: 228 crowdworkers were recruited for reliance-data collection, 145 completed that task; for evaluation, 170 were recruited and 98 completed the CC task.
- The reliance model used a Transformer encoder-based architecture trained on collaboration history; the task AI for CAPTCHA recognition used ResNet-18, and evaluation used ANCOVA on F-scores and subjective ratings.
Limitations
- The paper states that vPred-RC is not immediately applicable to tasks where dynamics during each trial matter, such as autonomous driving, and that it assumes task results can be categorized only as success or failure.
- The text makes evident that the experiments are limited to a CAPTCHA-based crowdworker setting with mostly Japanese participants; it also notes a post-hoc power analysis of 0.73 and that the reliance model still had failures due to poor prediction, indicating room for improvement.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 2 Not valid · 6 Not checked · 80 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, the F-score decreased as the number of RCCs decreased.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy is actually biased by the dataset used for the training.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The interaction effect was not statistically significant ( F (1 , 110) = 1 . 387; p = . 242; η 2 p = . 012 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, participants with a score of zero were concentrated in cases where the rate was 0.0 or 0.2.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in 8.0 percent of cases, the reliance model predicted that omitting an RCC instead would contribute to reducing the discrepancy.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
26 more claims
Create a free account to see all 92 claims and every source verdict.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 36
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615 · cited 1×
“We also conducted a post-hoc power analysis on the ANCOVA in subsection V-D using G*Power [33] and calculated a power (1 − β ) of 0.73.”
MethodInfluentialPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“However, the blackbox nature of ML models makes it difficult for end-users to understand the behavior of such agents [1], [2].”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“Particularly, a lack of understanding about what agents can and cannot do leads users to over-or under-rely on them [3], [4].”
BackgroundOn calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
CAPTCHA: Using Hard AI Problems for Security
von Ahn, L., Blum, M., Hopper, N. J., & Langford, J. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-39200-9_18 · Cited by 1,447 · cited 1×
“CAPTCHA is originally a task in which a human enters characters written in a noised and distorted image [19].”
BackgroundExplainable AI: From black box to glass box
Rai, A. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00710-5 · Cited by 1,317
Hoo, Z. H., Candlish, J., & Teare, D. (2017) · Emergency Medicine Journal · doi:10.1136/emermed-2017-206735 · Cited by 548
Overtrust of robots in emergency evacuation scenarios
Robinette, P., Li, W., Allen, R., Howard, A. M., & Wagner, A. R. (2016) · 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2016.7451740 · Cited by 468
Cited by
Göndöcs, D., Horváth, S., & Dörfler, V. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103622 · Cited by 52
Self-assessment in machines boosts human Trust
Warmsley, D., Choudhary, K., Rego, J., Viani, E., & Pilly, P. K. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2025.1557075 · Cited by 4
“Fukuchi and Yamada (2023) presented the Predictive Reliance Calibrator (Pred-RC) method to adaptively decide when to provide reliance calibration cues (confidence information), where reliance is considered an observable trust-related behavior.”
User Decision Guidance with Selective Explanation Presentation from Explainable-AI
Fukuchi, Y., & Yamada, S. (2024) · IEEE RO-MAN · doi:10.1109/ro-man60168.2024.10731188 · Cited by 6
“Another relevant study, Pred-RC [20], [21], aims to predict the effect of explanations of AI performance so that users can avoid over/under-reliance.”
BackgroundSelf-citationAn Identification and Analysis of Emotion Via Number of Visits and PR In Designing ML Models
Malhotra, S., Dhabalia, D., Ghobash, A., et al. (2024) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite60783.2024.10617396
“[2] II.”
Promoting Reliable Artificial Intelligence by Means of an Ethical Foundation for Trust Evaluation
Tewari, D., Abdullah, F. H., Jabbar, I., Al‐Farouni, M., Abdtawfeeq, T. H., & Alwan, A. S. (2024) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite60783.2024.10616529
Metadata
- Authors
- Fukuchi, Y., & Yamada, S.
- Affiliations
- National Institute of Informatics (JP)
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 138870-138881
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4389352627
- Cited by
- 16
- References
- 35
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 4 · 2023: 1
- FWCI
- 2.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Japan Science and Technology Agency (JST) Core Research for Evolutional Science and Technology (CREST), Japan (JPMJCR21D4)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- AI reliance, human-AI collaboration, cognitive models, human-AI communication, overreliance on AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Dynamic Selection of Reliance Calibration Cues With AI Reliance Model. https://app.cochat.ai/papers/10.1109/access.2023.3339548
- The paper
- Fukuchi, Y., & Yamada, S. (2023). Dynamic Selection of Reliance Calibration Cues With AI Reliance Model. IEEE Access. https://doi.org/10.1109/access.2023.3339548
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.