Cited by 248Human-Automation Interaction and SafetySocial Robot Interaction and HRIUsability and User Interface Design

The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.

Adaptive trust calibration for human-AI collaboration

Open access
Okamura, Kazuo, Yamada, Seiji·2020·PLOS ONE

Abstract

Safety and efficiency of human-AI collaboration often depend on how humans could appropriately calibrate their trust towards the AI agents. Over-trusting the autonomous system sometimes causes serious safety issues. Although many studies focused on the importance of system transparency in keeping proper trust calibration, the research in detecting and mitigating improper trust calibration remains very limited. To fill these research gaps, we propose a method of adaptive trust calibration that consists of a framework for detecting the inappropriate calibration status by monitoring the user's reliance behavior and cognitive cues called "trust calibration cues" to prompt the user to reinitiate trust calibration. We evaluated our framework and four types of trust calibration cues in an online experiment using a drone simulator. A total of 116 participants performed pothole inspection tasks by using the drone's automatic inspection, the reliability of which could fluctuate depending upon the weather conditions. The participants needed to decide whether to rely on automatic inspection or to do the inspection manually. The results showed that adaptively presenting simple cues could significantly promote trust calibration during over-trust.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes adaptive trust calibration: a behavior-based framework detects over-trust/under-trust and presents “trust calibration cues” (TCCs) only when needed. In an online drone-simulator experiment, adaptively presented TCCs significantly increased manual choices and improved sensitivity d′ during over-trust, whereas continuous reliability information alone did not.

Key findings

  • Of 194 online participants, 116 successfully inspected at least 15 checkpoints; they completed 1,740 inspections with a correct rate of 0.74, choosing automatic inspection 1,236 times (0.71) and manual inspection 504 times (0.29).
  • A two-factor mixed ANOVA found a significant main effect of weather period on manual rates [F(1, 111) = 51.69, p < 0.01] and a significant interaction between TCC group and period [F(4, 111) = 4.86, p < 0.01]. The NoTCC group did not significantly change manual rates between good and bad weather, while all TCC groups did: visual [F(1, 17) = 9.20, p < 0.01], audio [F(1, 21) = 5.54, p = 0.03], verbal [F(1, 28) = 62.9, p < 0.001], and anthropomorphic [F(1, 18) = 8.55, p < 0.01].
  • In bad weather, the verbal TCC group had a significantly higher manual rate than both NoTCC [t(111) = 4.77, adj.p < 0.01] and anthropomorphic TCC [t(111) = 2.74, adj.p = 0.04].
  • Sensitivity d′ in bad weather was significantly higher for the verbal group (0.92, SE 0.14) than for NoTCC [t(111) = 2.97, adj.p = 0.04, Cohen’s d = 0.75]; accuracy declined in bad weather but did not differ significantly among groups, though the verbal group had the highest mean accuracy (0.72, SE 0.04).

Methods

  • Online experiment using a 3D drone simulator built with CesiumJS and Bing Map API. Participants performed a pothole inspection task over 30 checkpoints (10 with potholes, 20 without), choosing between automatic inspection and manual inspection; they were randomly assigned to NoTCC, visual, audio, verbal, or anthropomorphic TCC groups.
  • Automatic inspection reliability Pauto was manipulated: good weather Pauto = 90% and d′ = 1.8; bad weather Pauto = 50% and d′ = 0.1. The framework detected over-trust by observing reliance behavior and comparing Pman and Pauto, then presented a TCC when over-trust was detected.
  • Main analyses were two-factor mixed ANOVAs with TCC group as between-subjects and checkpoint period as within-subjects, plus Holm–Bonferroni-adjusted post hoc comparisons and t-tests for Pman.

Limitations

  • Stated: the framework focuses only on performance-related factors, ignoring other automation benefits such as faster completion, lighter workload, or fewer risks; it evaluates only an over-trust scenario; it does not guarantee convergence of behavior; and it uses only a pothole inspection/reconnaissance task and four TCC types.
  • Stated: correct answers were not shown during the task, and automatic-inspection results were shown only when participants chose automatic inspection, not manual inspection; future work should test combinations of feedback information.
  • Evident: attrition was high—78 of 194 participants failed to complete the task within the time limit because they moved the drone away from the checkpoints—so the analyzed sample of 116 may not represent all recruited participants.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0229132

Cites

43 of 44 references verified1 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

0 of 22 cited sources checked · 0 supported · 0 flagged

Show all 47 claims

  • Probably not a substantial claim. 16
  • It may state two separate claims as one. 16
  • The quote only partly states this claim. 1
28 of 28
  • The authors state this as an assumption.

    Pauto fluctuates more widely than Pman under changing weather conditions.

    “Therefore, we assumed that Pauto would fluctuate more widely than Pman under changing weather conditions.”
  • The paper relies on this without support.

    Trust is a latent construct and cannot be directly measured; measuring trust experimentally is difficult in general.

    “Trust is a latent construct and cannot be directly measured; measuring trust experimentally is difficult in general.”
  • The paper relies on this without support.

    Users often fail to calibrate their trust in the agent and end up in over-trust or under-trust.

    “While an agent’s reliability changes for various reasons in an environment, the users often fail to calibrate their trust in the agent and end up in the status called over-trust or under-trust.”
  • The paper relies on this without support.

    Self-report measures are too intrusive to be viable in applied settings.

    “Most previous literature used self-report measures such as subjective rating with questionnaires [18] or rating scales [19, 20]; however, self-report measures are too intrusive to be viable in applied settings.”
  • The paper relies on this without support.

    Physiological and neural measures usually require special hardware.

    “Although these measures can be used for dynamic tracking of trust, they usually require special hardware.”
  • The paper relies on this without support.

    TCC rates are a valid indicator of over-trust or trust calibration status.

    “TCC rates in the verbal TCC group were higher in the early part of the period and gradually decreased. This indicates that over-trust decreased during this period.”
  • The authors state this as an assumption.

    Manual accuracy can be used to estimate Pman.

    “The results of the manual accuracy were consistent with our assumption to estimate Pman − Pauto in this experiment.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(1, 111) = 51.69reported p < .01 · recomputed p < .001

    “F(1, 111) = 51.69, p < .01”

  • ConsistentF(4, 111) = 4.86reported p < .01 · recomputed p = .001

    “F(4, 111) = 4.86, p < .01”

  • ConsistentF(1, 17) = 9.2reported p < .01 · recomputed p = .008

    “F(1, 17) = 9.20, p < .01”

  • ConsistentF(1, 21) = 5.54reported p = .03 · recomputed p = .028

    “F(1, 21) = 5.54, p = .03”

  • ConsistentF(1, 28) = 62.9reported p < .001 · recomputed p < .001

    “F(1, 28) = 62.9, p < .001”

  • ConsistentF(1, 18) = 8.55reported p < .01 · recomputed p = .009

    “F(1, 18) = 8.55, p < .01”

  • Consistentt(114) = -1.08reported p = .28 · recomputed p = .282

    “t(114) = −1.08, p = .28”

  • Consistentt(71) = -2.17reported p = .03 · recomputed p = .033

    “t(71) = −2.17, p = .03”

  • Consistentt(200) = 17.79reported p < .01 · recomputed p < .001

    “t(200) = 17.79, p < .01”

9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • CKP9 has nonzero TCC rates despite a stated no-presentation period

    Minor

    The text says no TCCs were presented from CKP7 through CKP9, but Table 1 reports nonzero TCC presentation rates at CKP9 for all four TCC groups.

1 of 1

Citations

Cites

Flagged first, then by citations

Show all 43 references

Research works printed but not linked: 1

  1. Adams BD, Bruyn LE, Houde S. Trust in automated systems literature review. Defence Research and Development Canada; 2003. Report No. CR-2003-096.

Cited by

Sorted by citations

Items 101–200 of 248

Metadata

Authors
Okamura, K., & Yamada, S.
Affiliations
  • The Graduate University for Advanced Studies, SOKENDAI (JP)
  • National Institute of Informatics (JP)
Corresponding author
Okamura, Kazuo
Venue
PLOS ONE
Published
2020-02-21
Volume
15
Issue
2
Pages
e0229132
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2954555759
Cited by
239
References
43
Citations by year
2026: 65 · 2025: 56 · 2024: 46 · 2023: 31 · 2022: 18 · 2021: 16 · 2020: 7
FWCI
19.67
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • The Japanese Society for Artificial Intelligence
Funders
Japan Society for the Promotion of Science (26118005)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
trust calibration, human-AI collaboration, cognitive cues, adaptive prompts, system transparency, trust in AI, drone inspection, human-AI decision-making
MeSH terms
Adult, Aged, Artificial Intelligence, Calibration, Female, Humans, Male, Man-Machine Systems, Middle Aged, Surveys and Questionnaires, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review