CoChat

Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation

Open access
Holland, Christopher, Perry, Grace, Neyedli, Heather F. · 2024 · Proceedings of the Human Factors and Ergonomics Society Annual Meeting

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 6 of the 13 sources its claims cite were checked for support.

Abstract

Trust and system reliability can influence a user's dependence on automated systems. This study aimed to investigate how increases and decreases in automation reliability affect users' trust in these systems and how these changes in trust are associated with users' dependence on the system. Participants completed a color identification task with the help of an automated aid, where the reliability of this aid either increased from 50% to 100% or decreased from 100% to 50% as the task progressed, depending on which group the participants were assigned to. Participants' trust, self-confidence, and dependence on the system were measured throughout the experiment. There were no differences in trust between the two groups throughout the experiment; however, participants' dependence behavior did follow system reliability. These findings highlight that trust is not always correlated with system reliability, and that although trust can often influence dependence, it does not always determine it.

Overview

CoChat Summary

## TL;DR

This study tested whether trust and dependence change as automation reliability increases or decreases from 50% to 100% across blocks in a color-identification task. Trust stayed flat in both groups, but dependence behavior tracked reliability, suggesting dependence can change independently of trust.

## Key findings

  • Trust did not differ by reliability direction or over time: no main effect of group on trust (*F*[1, 78] = 0.05, *p* = .82, η²p < .001), no main effect of block (*F*[3.59, 280.3] = 1.27, *p* = .29), and no interaction (*F*[3.59, 280.9] = 0.47, *p* = .74).
  • Dependence showed a significant group × block interaction (*F*[3.15, 245.4] = 6.21, *p* < .001, η²p = .07): in the increasing-reliability group the slope was 0.02 (SE = 0.01, *t* = 1.43, *p* = .15), and in the decreasing-reliability group it was −0.03 (SE = 0.01, *t* = −1.93, *p* = .05).
  • Self-confidence remained stable across group and block: no main effect of group (*F*[1, 78] = 0.69, *p* = .41), no main effect of block (*F*[3.65, 285.1] = 0.07, *p* = .99), and no interaction (*F*[3.65, 285.1] = 0.98, *p* = .41).
  • Final-response performance tracked reliability, with a significant group × block interaction (*F*[2.16, 168.1] = 29.0, *p* < .001, η²p = .27); initial-response performance showed no significant group effect (*F*[1, 78] = 0.79, *p* = .38), though the block-adjusted group effect approached the corrected alpha (*F*[4.73, 369.0] = 3.12, *p* = .01, η²p = .04).

## Methods

  • Design: 2-group experiment with changing automation reliability; one group experienced reliability increasing from 50% to 100% in 10% increments, the other decreasing from 100% to 50%, across six experimental blocks of 50 trials each after a 25-trial practice block.
  • Sample/data: 139 MTurk participants were recruited; after excluding datasets with initial-response performance below 56.7%, the analyzed sample was 80 participants (40 per condition).
  • Analysis: Trust, self-confidence, dependence behavior, and performance were analyzed using 2 (group) × 6 (block) mixed ANOVAs with Bonferroni correction; Greenhouse-Geisser corrections were applied when sphericity was violated.

## Limitations

  • The paper states that MTurk participants may limit generalizability to broader populations.
  • The paper states that the color-identification task may not fully capture the complexity of real-world automated systems.
  • Evident from the text: the study’s dependence measure is behavioral switching to agree with automation, so it captures one operationalization of dependence rather than all aspects of dependence.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/10711813241277531

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Progression of experimental trial. Slide 1: stimulus. Slide 2: initial color selection. Slide 3: Automation recommendation with option to change initial selection (original selection is also provided). Slide 4 and 5 (right side of figure) are the Trust and Self-Confidence questions, presented every 10 trials.
Figure 1. Progression of experimental trial. Slide 1: stimulus. Slide 2: initial color selection. Slide 3: Automation recommendation with option to change initial selection (original selection is also provided). Slide 4 and 5 (right side of figure) are the Trust and Self-Confidence questions, presented every 10 trials.CC BY · PubMed Central
Figure 2. Dependence behavior by block. Dependence was scored on a 0 to 1 scale as the proportion of times participants deviated from their initial response to match the automation’s recommendation. In all results figures, individual points represent a participant’s mean for a given block. The upper and lower limits of the ribbon represent 1 standard error above and below the mean for a given block and condition.
Figure 2. Dependence behavior by block. Dependence was scored on a 0 to 1 scale as the proportion of times participants deviated from their initial response to match the automation’s recommendation. In all results figures, individual points represent a participant’s mean for a given block. The upper and lower limits of the ribbon represent 1 standard error above and below the mean for a given block and condition.CC BY · PubMed Central
Figure 3. Self-reported trust in the automation and self-confidence across blocks. Both trust and self-confidence were scored on a scale of 0 to 100.
Figure 3. Self-reported trust in the automation and self-confidence across blocks. Both trust and self-confidence were scored on a scale of 0 to 100.CC BY · PubMed Central
Figure 4. Performance for participants’ initial and final responses (given after automation recommendation) across blocks. Performance was scored on a 0 to 1 scale as the proportion of correct responses in a block.
Figure 4. Performance for participants’ initial and final responses (given after automation recommendation) across blocks. Performance was scored on a 0 to 1 scale as the proportion of correct responses in a block.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 0 Not valid · 5 Not checked · 31 Own results and interpretations

17 more claims

Create a free account to see all 42 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 8
  • It may state two separate claims as one. 13
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 18 references

Research works printed but not linked: 1

  1. Clark J. H. (1924). The Ishihara test for color blindness. American Journal of Physiological Optics, 5, 269–276.

Cited by

See all 8

Metadata

Authors
Holland, C., Perry, G., & Neyedli, H. F.
Affiliations
Dalhousie University (CA)
Corresponding author
Holland, Christopher
Venue
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
Published
2024-09
Volume
68
Issue
1
Pages
604-610
ISSN
  • 0163-5182 (ISSN-L)
  • 1071-1813
  • 2169-5067
OpenAlex
W4402136577
Cited by
8
References
16
Citations by year
2026: 5 · 2025: 3
FWCI
3.79
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the Human Factors and Ergonomics Society Annual Meeting (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Innovation for Defence Excellence and Security
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Risk and Safety Analysis
Keywords
trust in automation, system reliability, self-confidence, automated decision aids

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation. https://app.cochat.ai/papers/10.1177/10711813241277531
The paper
Holland, C., Perry, G., & Neyedli, H. F. (2024). Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. https://doi.org/10.1177/10711813241277531

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.