Automation Error Type and Methods of Communicating Automation Reliability Affect Trust and Performance: An Empirical Study in the Cyber Domain
The DOI matches its Crossref record; 59 of the 60 references it cites were found in a registry or an index.
Abstract
Antiphishing aid systems, among other automated systems, are not perfectly reliable. Automated systems can make errors, thereby resulting in false alarms or misses. An automated system's capabilities need to be communicated to the users to maintain proper user trust. System capabilities can be learned through an explicit description or from experience. Using a phishing-detection system as a testbed in this article, we systematically varied automation error type and the method of communicating system reliability in a factorial design and measured their effects on human performance and trust in the automation. Participants were asked to classify emails as legitimate or phishing with assistance from the phishing-detection system. The results from 510 participants suggest that learning through experience with feedback improved trust calibration for both objective and subjective trust measures in most conditions. Moreover, false alarms lowered trust more than misses for both unreliable and reliable systems, and false alarms turned out to be beneficial for proper trust calibration when using unreliable systems. Design implications of the results include using feedback whenever possible and choosing false alarms over misses for unreliable systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many argued that such behavioral measures displayed a more objective trust measure than subjective self-reports (see, e.g., [14]).”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [14]
In the citing paper
“For example, Chen et al. [14] found that false alarms reduced perceived trustworthiness more than misses.”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [14]
In the citing paper
“Yet, others found positive efects of feedback [14,15,94].”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [14]
Citations
Cites
(1966) · Medical Entomology and Zoology · Cited by 10,293
Signal detection theory and psychophysics
Ingleby, J. D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,509
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Dhamija, R., Tygar, J. D., & Hearst, M. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124861 · Cited by 1,376
Do multiple outcome measures require p-value adjustment?
Feise, R. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-8 · Cited by 1,299
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Trust, self-confidence, and operators' adaptation to automation
Lee, J. D., & Moray, N. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1007 · Cited by 932
Multivariate analysis versus multiple univariate analyses.
Huberty, C. J., & Morris, J. D. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.302 · Cited by 930
The description–experience gap in risky choice
Hertwig, R., & Erev, I. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.09.004 · Cited by 911
Corrections for extreme proportions and their biasing effects on estimated values ofd′
Hautus, M. J. (1995) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203619 · Cited by 894
MUIR, B. M. (1994) · Ergonomics · doi:10.1080/00140139408964957 · Cited by 873
Usability engineering methods for software developers
Holzinger, A. (2005) · Communications of the ACM · doi:10.1145/1039539.1039541 · Cited by 855
Schaefer, K. E., Chen, J. Y. C., Szalma, J. L., & Hancock, P. A. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816634228 · Cited by 768
Self-citationSmall feedback‐based decisions and their limited correspondence to description‐based decisions
Barron, G., & Erev, I. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.443 · Cited by 735
You've been warned: an empirical study of the effectiveness of web browser phishing warnings
Egelman, S., Cranor, L. F., & Hong, J. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357219 · Cited by 607
Hong, J. (2012) · Communications of the ACM · doi:10.1145/2063176.2063197 · Cited by 551
Anti-Phishing Phil: the design and evaluation of a game that teaches people not to fall for phish
Sheng, S., Magnien, B., Kumaraguru, P., Acquisti, A., Cranor, L. F., Hong, J., & Nunge, E. (2007) · Proceedings of the 3rd symposium on Usable privacy and security · doi:10.1145/1280680.1280692 · Cited by 509
Low target prevalence is a stubborn source of errors in visual search tasks.
Wolfe, J. M., Horowitz, T. S., Van Wert, M. J., Kenner, N. M., Place, S. S., & Kibbi, N. (2007) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.136.4.623 · Cited by 418
Kim, J., & Moon, J. Y. (1998) · Interacting with Computers · doi:10.1016/s0953-5438(97)00037-4 · Cited by 402
Teaching Johnny not to fall for phish
Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F., & Hong, J. (2010) · ACM Transactions on Internet Technology · doi:10.1145/1754393.1754396 · Cited by 396
Adaptive automation, trust, and self-confidence in fault management of time-critical tasks.
Moray, N., Inagaki, T., & Itoh, M. (2000) · Journal of Experimental Psychology: Applied · doi:10.1037//1076-898x.6.1.44 · Cited by 345
A choice prediction competition: Choices from experience and from description
Erev, I., Ert, E., Roth, A. E., Haruvy, E., Herzog, S. M., Hau, R., Hertwig, R., Stewart, T., West, R., & Lebiere, C. (2010) · Journal of Behavioral Decision Making · doi:10.1002/bdm.683 · Cited by 329
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301
Effects of Imperfect Automation on Decision Making in a Simulated Command and Control Task
Rovira, E., McGarry, K., & Parasuraman, R. (2007) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007779598082 · Cited by 279
A meta-analytic review of two modes of learning and the description-experience gap.
Wulff, D. U., Mergenthaler-Canseco, M., & Hertwig, R. (2018) · Psychological Bulletin · doi:10.1037/bul0000115 · Cited by 271
Impact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013) · 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2013.6483596 · Cited by 234
Department of Defense Dictionary of Military and Associated Terms
Gortney, W. E. (2016) · Cited by 228
McGuirl, J. M., & Sarter, N. B. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166334 · Cited by 212
Systems with Human Monitors: A Signal Detection Analysis
Sorkin, R. D., & Woods, D. D. (1985) · Human–Computer Interaction · doi:10.1207/s15327051hci0101_2 · Cited by 211
Redmiles, E. M., Kross, S., & Mazurek, M. L. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00014 · Cited by 187
An evaluation of a multiple interface design solution for bloated software
McGrenere, J., Baecker, R. M., & Booth, K. S. (2002) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503406 · Cited by 169
Reducing false positives in intrusion detection systems
Spathoulas, G. P., & Katsikas, S. K. (2009) · Computers & Security · doi:10.1016/j.cose.2009.07.008 · Cited by 148
Chancey, E. T., Bliss, J. P., Yamani, Y., & Handley, H. A. H. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816682648 · Cited by 144
Getty, D. J., Swets, J. A., Pickett, R. M., & Gonthier, D. (1995) · Journal of Experimental Psychology Applied · doi:10.1037/1076-898x.1.1.19 · Cited by 142
Impact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A. M., & Yanco, H. A. (2013) · doi:10.5555/2447556.2447663 · Cited by 139
Quantifying Phishing Susceptibility for Detection and Behavior Decisions
Canfield, C. I., Fischhoff, B., & Davis, A. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816665025 · Cited by 129
Proctor, R. W., & Chen, J. (2015) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815585906 · Cited by 94
Self-citationUnderstanding reliance on automation: effects of error type, error distribution, age and experience
Sanchez, J., Rogers, W. A., Fisk, A. D., & Rovira, E. (2014) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2011.611269 · Cited by 92
The Role of Psychology in Enhancing Cybersecurity
Wiederhold, B. K. (2014) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2014.1502 · Cited by 85
Don't work. Can't work? Why it's time to rethink security warnings
Krol, K., Moroz, M., & Sasse, M. A. (2012) · 2012 7th International Conference on Risks and Security of Internet and Systems (CRiSIS) · doi:10.1109/crisis.2012.6378951 · Cited by 81
Training users to counteract phishing
Mayhorn, C. B., & Nyeste, P. G. (2012) · Work · doi:10.3233/wor-2012-1054-3549 · Cited by 64
Chen, J., Mishler, S., Hu, B., Li, N., & Proctor, R. W. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.05.010 · Cited by 50
Self-citationUnmanned Aerial Vehicle Flight Control: False Alarms versus Misses
Dixon, S. R., Wickens, C. D., & Chang, D. (2004) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120404800133 · Cited by 44
The dynamics of trust in cyberdomains
Hoffman, R. R., Lee, J. D., Woods, D. D., Shadbolt, N., Miller, J., & Bradshaw, J. M. (2009) · IEEE Intelligent Systems · doi:10.1109/mis.2009.124 · Cited by 43
Use of Phishing Training to Improve Security Warning Compliance: Evidence from a Field Experiment
Yang, W., Xiong, A., Chen, J., Proctor, R. W., & Li, N. (2017) · Proceedings of the Hot Topics in Science of Security: Symposium and Bootcamp · doi:10.1145/3055305.3055310 · Cited by 43
Self-citationHuegli, D., Merks, S., & Schwaninger, A. (2020) · Applied Ergonomics · doi:10.1016/j.apergo.2020.103094 · Cited by 43
Pak, R., Rovira, E., McLaughlin, A. C., & Baldwin, N. (2017) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2016.1175523 · Cited by 39
Embedding Training Within Warnings Improves Skills of Identifying Phishing Webpages
Xiong, A., Proctor, R. W., Yang, W., & Li, N. (2018) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720818810942 · Cited by 36
Trust and privacy in the future internet—a research perspective
Rooy, D. V., & Bus, J. C. P. (2010) · Identity in the Information Society · doi:10.1007/s12394-010-0058-7 · Cited by 34
Something Smells Phishy: Exploring Definitions, Consequences, and Reactions to Phishing
Kelley, C. M., Hong, K. W., Mayhorn, C. B., & Murphy-Hill, E. R. (2012) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181312561447 · Cited by 33
Empirically Valid Principles of Training
Healy, A. F., Schneider, V. I., & Bourne, L. E. (2012) · doi:10.4324/9780203816783-5 · Cited by 21
An evaluation of a multiple interface design solution for bloated software
McGrenere, J., Baecker, R. M., & Booth, K. S. (2002) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/503403.503406 · Cited by 20
Vieira, M. M., Ugrinowitsch, H., Oliveira, F. S., Gallo, L. G., & Benda, R. N. (2012) · Perceptual and Motor Skills · doi:10.2466/22.23.pms.115.5.360-369 · Cited by 8
Description-Experience Gap: The Role of Feedback and Description in Human Trust in Automation
Mishler, S., Chen, J., Šabić, E., Hu, B., Li, N., & Proctor, R. W. (2017) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601559 · Cited by 4
Self-citation
Research works printed but not linked: 1
- parrish, 2009, A personality based model for determining susceptibility to phishing attacks, Little Rock University of Arkansas, 285
Cited by
Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 144
“; Our set of future research directions results from a structured summary of our analysis based on the implications of the approaches (definitions, methods, tasks and measures) adopted by the authors to foster appropriate trust in human-AI interaction.”
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“Yet, others found positive efects of feedback [14, 15, 94].”
BackgroundPoornikoo, M., Gyldensten, W., Vesin, B., & Øvergård, K. I. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2399439 · Cited by 36
“…findings from the experimental study on Maritime Autonomous Surface Ships (MASS) align with the established research (eg, Chavaillaz et al., 2016; Chen et al., 2021; Hussein et al., 2020; Lee & Moray, 1994; Moray et al., 2000; Yang et al., 2016) suggesting that trust in automation is closely…”
Automation Bias and Complacency in Security Operation Centers
Tilbury, J., & Flowerday, S. (2024) · Computers · doi:10.3390/computers13070165 · Cited by 34
“Ten of the 13 represent core papers, as they clearly identify the implications that automation has on security analysts, ranging from bias, complacency, mis-and disuse, human-machine trust, and trust intentions [3,6,14,28,36,37,39,56,57].”
BackgroundMethodInfluentialHumans and Automation: Augmenting Security Operation Centers
Tilbury, J., & Flowerday, S. (2024) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp4030020 · Cited by 30
“The prevalence of automated cybersecurity tools should provide ample motivation toward studying human-automation interaction, yet research evaluating trust in these tools appears limited [9].”
Background
Metadata
- Authors
- Chen, J., Mishler, S., & Hu, B.
- Affiliations
- Old Dominion University (US)
- Venue
- IEEE Transactions on Human-Machine Systems
- Published
- 2021-10
- Volume
- 51
- Issue
- 5
- Pages
- 463-473
- ISSN
- 2168-2291 (ISSN-L)
- 2168-2305
- OpenAlex
- W3130493744
- Cited by
- 40
- References
- 60
- Citations by year
- 2026: 10 · 2025: 6 · 2024: 13 · 2023: 7 · 2022: 3 · 2021: 1
- FWCI
- 5.88
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Human-Machine Systems (publishedVersion)
- Funders
- National Science Foundation (1760347)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- trust in automation, false alarms, Miss, phishing detection, trust calibration, feedback learning, human performance, email classification, human-automation interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.