Automation failures on tasks easily performed by operators undermine trust in automated aids
Abstract
Objective: We tested the hypothesis that automation errors on tasks easily performed by humans undermine trust in automation. Background: Research has revealed that the reliability of imperfect automation is frequently misperceived. We examined the manner in which the easiness and type of imperfect automation errors affect trust and dependence. Method: Participants performed a target detection task utilizing an automated aid. In Study 1, the aid missed targets either on easy trials (easy miss group) or on difficult trials (difficult miss group). In Study 2, we manipulated both easiness and type of error (miss vs. false alarm). The aid erred on either difficult trials alone (difficult errors group) or on difficult and easy trials (easy miss group; easy false alarm group). Results: In both experiments, easy errors led to participants mistrusting and disagreeing more with the aid on difficult trials, as compared with those using aids that generated only difficult errors. This resulted in a downward shift in decision criterion for the former, leading to poorer overall performance. Misses and false alarms led to similar effects. Conclusion: Automation errors on tasks that appear “easy” to the operator severely degrade trust and reliance. Application: Potential applications include the implementation of system design solutions that circumvent the negative effects of easy automation errors.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lack of transparency may lead humans to overreact to AI errors, inferring AI is generally unreliable and incompetent, leading to mistrust and misuse.
“Third, the lack of transparency may lead humans to overreact to AI errors: if humans cannot understand the cause of an AI error or if they consider it anomalous compared to typical human errors, they may be tempted to infer that the AI is generally unreliable and incompetent, leading to mistrust and misuse of the AI system itself [71 , 72] .”
From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [71]
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2× · see citing works
“…regarding the functional limitations of automation (i.e., informing operators about reasons a machine might err; see Dzindolet et al., 2001; Lee & See, 2004) will make automation more “trustable” and give operators a better opportunity to appropriately discriminate between the relative…”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1× · see citing works
“The availability of decision-aiding technology in recent times has provided operators with automated diagnostic aids that help perform both of these stages of information processing (Parasuraman, Sheridan, & Wickens, 2000).”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · see citing works
MUIR, B. M. (1994) · Ergonomics · doi:10.1080/00140139408964957 · Cited by 873 · cited 1× · see citing works
“Nonetheless, the relationship between trust and operators’ reliance upon imperfectly reliable automation is often complex and multifaceted (e.g., Lee & Moray, 1992, 1994; Lerch, Prietula, & Kulik, 1997; Muir, 1987, 1994; Sheridan, 2002; Wiegmann, Rich, & Zhang, 2001).”
BackgroundThe Perceived Utility of Human and Automated Aids in a Visual Detection Task
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494856 · Cited by 386 · see citing works
Alarm effectiveness in driver-centred collision-warning systems
PARASURAMAN, R., HANCOCK, P. A., & OLOFINBOBA, O. (1997) · Ergonomics · doi:10.1080/001401397188224 · Cited by 205 · see citing works
Automated diagnostic aids: The effects of aid reliability on users' trust and reliance
Wiegmann, D. A., Rich, A., & Zhang, H. (2001) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110110306 · Cited by 176 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817 · see citing works
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · see citing works
Ho, A., Hancock, J., & Miner, A. S. (2018) · Journal of Communication · doi:10.1093/joc/jqy026 · Cited by 629 · see citing works
“On the other hand, people assume that chatbots are worse at emotional tasks than humans (Madhavan, Wiegmann, & Lacson, 2006), which may negatively impact emotional disclosure with chatbots.”
BackgroundAlmost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614 · see citing works
A Systematic Review of Patient Acceptance of Consumer Health Information Technology
Or, C. K. L., & Karsh, B.-T. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2888 · Cited by 586 · see citing works
Trusting Automation: Designing for Responsivity and Resilience
Chiou, E. K., & Lee, J. D. (2023) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208211009995 · Cited by 301 · see citing works
Manzey, D., Reichenbach, J., & Onnasch, L. (2012) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343411433844 · Cited by 277 · see citing works
Complacency and Automation Bias in the Use of Imperfect Automation
Wickens, C. D., Clegg, B. A., Vieane, A. Z., & Sebok, A. L. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815581940 · Cited by 176 · see citing works
Use, disuse, and misuse of automated refactorings
Vakilian, M., Chen, N., Negara, S., Rajkumar, B. A., Bailey, B. P., & Johnson, R. E. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227190 · Cited by 106 · see citing works
Kim, T., & Song, H. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2021.101595 · Cited by 101 · see citing works
Chen, Y., Zahedi, F. M., Abbasi, A., & Dobolyi, D. (2021) · Information & Management · doi:10.1016/j.im.2020.103394 · Cited by 53 · see citing works
Chen, J., Mishler, S., Hu, B., Li, N., & Proctor, R. W. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.05.010 · Cited by 50 · see citing works
Enhancing human-AI collaboration: The case of colonoscopy
Introzzi, L., Zonca, J., Cabitza, F., Cherubini, P., & Reverberi, C. (2024) · Digestive and Liver Disease · doi:10.1016/j.dld.2023.10.018 · Cited by 28 · see citing works
Kox, E. S., Siegling, L. B., & Kerstholt, J. H. (2022) · International Journal of Social Robotics · doi:10.1007/s12369-022-00871-4 · Cited by 22 · see citing works
Continuous Calibration of Trust in Automated Systems
Merritt, S. M., Huber, K., LaChapell-Unnerstall, J., & Lee, D. (2014) · doi:10.21236/ada606748 · Cited by 20 · see citing works
Kim, J. Y., Lester, C., & Yang, X. J. (2025) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208251326795 · Cited by 21 · see citing works
- Authors
- Madhavan, P., Wiegmann, D. A., & Lacson, F. C.
- Affiliations
- Carnegie Mellon University (US)
- University of Illinois Urbana-Champaign (US)
- ORCID
- Wiegmann, Douglas A.
- Venue
- Human Factors: The Journal of the Human Factors and Ergonomics Society
- Publisher
- SAGE Publications
- Published
- 2006-06
- Volume
- 48
- Issue
- 2
- Pages
- 241-256
- ISSN
- 0018-7208 (ISSN-L)
- 1547-8181
- OpenAlex
- W1984361612
- Cited by
- 268
- References
- 8
- Citations by year
- 2026: 19 · 2025: 21 · 2024: 23 · 2023: 20 · 2022: 16 · 2021: 19 · 2020: 15 · 2019: 19 · 2018: 14 · 2017: 13 · 2016: 13 · 2015: 9 · 2014: 12 · 2013: 8 · 2012: 9
- FWCI
- 8.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Factors The Journal of the Human Factors and Ergonomics Society (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Risk and Safety Analysis
- Keywords
- trust in automation, human-automation interaction, target detection, false alarms
- MeSH terms
- Automation, Equipment Failure, Humans, Illinois, Task Performance and Analysis, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
