Feedback on system or operator performance: Which is more useful for the timely detection of changes in reliability, trust calibration and appropriate automation usage?
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Creating safe human-machine systems requires that operators can quickly notice changes in system reliability in the interest of trust calibration and proper automation usage. Operators’ readiness to trust a system is determined not only by the performance of the automation but also by their confidence in their own abilities. This study therefore compared the usefulness of feedback on the performance of either agent. The experiment required two groups of ten participants each to perform an automation-assisted target identification task with “Automation Performance Feedback” (APF) or “Operator Performance Feedback” (OPF). Four different scenarios differed with respect to the degree and duration of changes in system reliability. Findings indicate that APF was more effective for supporting timely adjustments of perceived system reliability, especially with large and long reliability changes. Subjective trust ratings and performance were not affected, however, suggesting that these two factors are closely linked and more relevant for automation reliance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“Here, trust is defined as “the attitude that an agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability” (Lee & See, 2004).”
BackgroundA Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., & Parasuraman, R. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811417254 · Cited by 2,051 · cited 1×
“A considerable number of automation, human-and environment-related factors have been shown to affect people’s trust in, and usage of modern technologies (Hancock et al., 2011).”
BackgroundTrust, self-confidence, and operators' adaptation to automation
Lee, J. D., & Moray, N. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1007 · Cited by 932 · cited 2×
“However, at the same time, inappropriate usage of the automation has been observed, including both misuse and disuse (Lee & Moray, 1994).”
BackgroundTrust and etiquette in high-criticality automated systems
Parasuraman, R., & Miller, C. A. (2004) · Communications of the ACM · doi:10.1145/975817.975844 · Cited by 311 · cited 1×
“However, the effect of system reliability is mediated by a human-related factor, namely how people perceive their own abilities and the difficulty of the task (Parasuraman & Miller, 2004).”
BackgroundAffective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260 · cited 1×
“Trust propensity was collected once before the start of the experiment, using Merritt’s (2011) questionnaire.”
MethodMcGuirl, J. M., & Sarter, N. B. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166334 · Cited by 212 · cited 1×
“For example, a study conducted by McGuirl & Sarter (2006) found that operators’ trust in automation was more appropriate when dynamic system confidence information was presented.”
BackgroundSelf-citationEffects of changing reliability on trust of robot systems
Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., & Yanco, H. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157702 · Cited by 188
Automated diagnostic aids: The effects of aid reliability on users' trust and reliance
Wiegmann, D. A., Rich, A., & Zhang, H. (2001) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110110306 · Cited by 176 · cited 1×
“For example, overall, people have a tendency to underestimate the actual capability of the automation when its performance is not perfect (Wiegmann, Rich & Zhang, 2001).”
BackgroundMadhavan, P., & Wiegmann, D. A. (2007) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007x230154 · Cited by 157 · cited 1×
“Also, Madhavan and Wiegmann (2007) showed that trust in automation degraded when a system committed errors while detecting what was considered an easily identifiable target.”
BackgroundUser Trust Dynamics: An Investigation Driven by Differences in System Performance
Yu, K., Berkovsky, S., Taib, R., Conway, D., Zhou, J., & Chen, F. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025219 · Cited by 90 · cited 1×
“Other studies (e.g., Yu et al., 2017) emphasized feedback on operator performance.”
Background
Cited by
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“To achieve an alignment after an interaction, some studies provide feedback about the user and system performance (see e.g., [5, 14, 82]).”
BackgroundInfluentialCalibrating Reliance on Automated Advice: Transparency and Trust Calibration Feedback
Tatasciore, M., & Loft, S. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2487861 · Cited by 20
“In contrast to most prior studies (e.g., Chen et al., 2021; Lu & Sarter, 2019), trust calibration feedback in the current study provided information about both human and automation performance (Okamura & Yamada, 2020).”
Følstad, A., Law, E. L.-C., & van As, N. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3690383 · Cited by 17
Radensky, M., Séguin, J. A., Lim, J. S., Olson, K., & Geiger, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594043 · Cited by 14
Xu, S., Gong, Z. H., Li, Y., Zhao, Y., & Ahsan, A. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2719173
Metadata
- Authors
- Lu, Y., & Sarter, N.
- Affiliations
- University of Michigan (US)
- Venue
- Proceedings of the Human Factors and Ergonomics Society Annual Meeting
- Publisher
- SAGE Publications
- Published
- 2019-11
- Volume
- 63
- Issue
- 1
- Pages
- 312-316
- ISSN
- 0163-5182 (ISSN-L)
- 1071-1813
- 2169-5067
- OpenAlex
- W2991339092
- Cited by
- 5
- References
- 14
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 2
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Proceedings of the Human Factors and Ergonomics Society Annual Meeting (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Risk and Safety Analysis
- Keywords
- trust calibration, automation reliance, human-machine systems, operator confidence, target identification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Feedback on system or operator performance: Which is more useful for the timely detection of changes in reliability, trust calibration and appropriate automation usage? https://app.cochat.ai/papers/10.1177/1071181319631345
- The paper
- Lu, Y., & Sarter, N. (2019). Feedback on system or operator performance: Which is more useful for the timely detection of changes in reliability, trust calibration and appropriate automation usage? Proceedings of the Human Factors and Ergonomics Society Annual Meeting. https://doi.org/10.1177/1071181319631345
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.