Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 6 of the 13 sources its claims cite were checked for support.
Abstract
Trust and system reliability can influence a user's dependence on automated systems. This study aimed to investigate how increases and decreases in automation reliability affect users' trust in these systems and how these changes in trust are associated with users' dependence on the system. Participants completed a color identification task with the help of an automated aid, where the reliability of this aid either increased from 50% to 100% or decreased from 100% to 50% as the task progressed, depending on which group the participants were assigned to. Participants' trust, self-confidence, and dependence on the system were measured throughout the experiment. There were no differences in trust between the two groups throughout the experiment; however, participants' dependence behavior did follow system reliability. These findings highlight that trust is not always correlated with system reliability, and that although trust can often influence dependence, it does not always determine it.
Overview
CoChat Summary
## TL;DR
This study tested whether trust and dependence change as automation reliability increases or decreases from 50% to 100% across blocks in a color-identification task. Trust stayed flat in both groups, but dependence behavior tracked reliability, suggesting dependence can change independently of trust.
## Key findings
- Trust did not differ by reliability direction or over time: no main effect of group on trust (*F*[1, 78] = 0.05, *p* = .82, η²p < .001), no main effect of block (*F*[3.59, 280.3] = 1.27, *p* = .29), and no interaction (*F*[3.59, 280.9] = 0.47, *p* = .74).
- Dependence showed a significant group × block interaction (*F*[3.15, 245.4] = 6.21, *p* < .001, η²p = .07): in the increasing-reliability group the slope was 0.02 (SE = 0.01, *t* = 1.43, *p* = .15), and in the decreasing-reliability group it was −0.03 (SE = 0.01, *t* = −1.93, *p* = .05).
- Self-confidence remained stable across group and block: no main effect of group (*F*[1, 78] = 0.69, *p* = .41), no main effect of block (*F*[3.65, 285.1] = 0.07, *p* = .99), and no interaction (*F*[3.65, 285.1] = 0.98, *p* = .41).
- Final-response performance tracked reliability, with a significant group × block interaction (*F*[2.16, 168.1] = 29.0, *p* < .001, η²p = .27); initial-response performance showed no significant group effect (*F*[1, 78] = 0.79, *p* = .38), though the block-adjusted group effect approached the corrected alpha (*F*[4.73, 369.0] = 3.12, *p* = .01, η²p = .04).
## Methods
- Design: 2-group experiment with changing automation reliability; one group experienced reliability increasing from 50% to 100% in 10% increments, the other decreasing from 100% to 50%, across six experimental blocks of 50 trials each after a 25-trial practice block.
- Sample/data: 139 MTurk participants were recruited; after excluding datasets with initial-response performance below 56.7%, the analyzed sample was 80 participants (40 per condition).
- Analysis: Trust, self-confidence, dependence behavior, and performance were analyzed using 2 (group) × 6 (block) mixed ANOVAs with Bonferroni correction; Greenhouse-Geisser corrections were applied when sphericity was violated.
## Limitations
- The paper states that MTurk participants may limit generalizability to broader populations.
- The paper states that the color-identification task may not fully capture the complexity of real-world automated systems.
- Evident from the text: the study’s dependence measure is behavioral switching to agree with automation, so it captures one operationalization of dependence rather than all aspects of dependence.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
6 Valid · 0 Not valid · 5 Not checked · 31 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to expectations, trust in the automation decreased in both groups.”
§ Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation > Introduction
What the sources say
- Supportedcites Rittenberg et al. (2024) source
The Experiment 1 Discussion states that trust decreased in both the increasing- and decreasing-reliability groups, contrary to the authors’ hypothesis for the increasing group. This finding applies to Experiment 1; the Experiment 2 Discussion reports increased trust in a separate group that started at 70% reliability. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Trust can rapidly change with a single interaction (Chung & Yang, 2024; Yang et al., 2023).”
§ Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation > Introduction
What the sources say
- Not checked — no full text of the sourcecites Chung & Yang, 2024 source
- Supportedcites Yang et al., 2023 source
The Abstract (Conclusion) states that operators adjust their trust as a result of moment-to-moment interactions with automation. The Method (Measures) reports trust after each trial and calculates the change between trials, supporting the claim that trust can change after a single interaction. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unlike Rittenberg et al. (2024), there was no change in trust with changing reliability level in either group.”
§ Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation > Discussion
What the sources say
- Supportedcites Rittenberg et al. (2024) source
The citation marks a contrast: the citing study found no change, unlike Rittenberg et al. The source’s Results > ANOVA Descriptive Analysis reports significant changes in trust in both its increasing- and decreasing-reliability groups. The no-change finding is not attributed to Rittenberg et al. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no main effect of block on participants final response performance (F [2.16, 168.1] = 1.88, p = .15, η2p = .02).”
§ Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation > Results > Performance—Initial Response and Final Response
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the decreasing condition, the slope was estimated at −0.03 (SE = 0.01, t = −1.93, p = .05).”
§ Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation > Results > Dependence
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
17 more claims
Create a free account to see all 42 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554
Conceptual Issues in the Study of Dynamic Hazard Warnings
Meyer, J. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.2.196.37335 · Cited by 214
Effects of changing reliability on trust of robot systems
Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., & Yanco, H. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157702 · Cited by 188
Du, N., Haspiel, J., Zhang, Q., Tilbury, D., Pradhan, A. K., Yang, X. J., & Robert, L. P. (2019) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2019.05.025 · Cited by 167
Chancey, E. T., Bliss, J. P., Yamani, Y., & Handley, H. A. H. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816682648 · Cited by 144
Yang, X. J., Schemanske, C., & Searle, C. (2023) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208211034716 · Cited by 132
Merritt, S. M., Lee, D., Unnerstall, J. L., & Huber, K. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814561675 · Cited by 83
Du, N., Huang, K. Y., & Yang, X. J. (2020) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720819862916 · Cited by 55
Measures of Reliance and Compliance in Aided Visual Scanning
Meyer, J., Wiczorek, R., & Günzler, T. (2014) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720813512865 · Cited by 53
Research works printed but not linked: 1
- Clark J. H. (1924). The Ishihara test for color blindness. American Journal of Physiological Optics, 5, 269–276.
Cited by
Development and validation of a scale assessing perceived trustworthiness in large language models
Yankouskaya, A., Barajeeh, B., Babiker, A., Alshakhsi, S. A. A., Ma, Y. T., Ho, C. S. M., & Ali, R. (2026) · Discover Artificial Intelligence · doi:10.21203/rs.3.rs-7738215/v1 · Cited by 5
Tan, C., Xu, X., & Liang Ma, A. (2026) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2026.103924 · Cited by 7
Holland, C. W., & Neyedli, H. F. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251367370 · Cited by 2
“…did not change with reliability level, performance did change with automation reliability level suggesting that participants were relying on the automation throughout the experiment indicating a complacency effect (see also Holland et al., 2024, who directly measured dependence on the automation).”
Self-citationFrom AI literacy to cognitive AI safety: Rethinking safe AI use in education.
Zainuddin, Z., Churiyah, M., Xiaoyu, W., & Halili, S. H. (2026) · Interactive Technology and Smart Education · doi:10.1108/itse-12-2025-0396 · Cited by 1
Tan, C., Xu, X., & Ma, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5995607
Metadata
- Authors
- Holland, C., Perry, G., & Neyedli, H. F.
- Affiliations
- Dalhousie University (CA)
- Corresponding author
- Holland, Christopher
- ORCID
- Neyedli, Heather F.
- Venue
- Proceedings of the Human Factors and Ergonomics Society Annual Meeting
- Publisher
- SAGE Publications
- Published
- 2024-09
- Volume
- 68
- Issue
- 1
- Pages
- 604-610
- ISSN
- 0163-5182 (ISSN-L)
- 1071-1813
- 2169-5067
- PMC
- PMC11655268
- OpenAlex
- W4402136577
- Cited by
- 8
- References
- 16
- Citations by year
- 2026: 5 · 2025: 3
- FWCI
- 3.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (vor)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Proceedings of the Human Factors and Ergonomics Society Annual Meeting (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Innovation for Defence Excellence and Security
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Risk and Safety Analysis
- Keywords
- trust in automation, system reliability, self-confidence, automated decision aids
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation. https://app.cochat.ai/papers/10.1177/10711813241277531
- The paper
- Holland, C., Perry, G., & Neyedli, H. F. (2024). Calibrating Trust, Reliance and Dependence in Variable-Reliability Automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. https://doi.org/10.1177/10711813241277531
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.