The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Automation bias and verification complexity: a systematic review
Open accessAbstract
Introduction: While potentially reducing decision errors, decision support systems can introduce new types of errors. Automation bias (AB) happens when users become overreliant on decision support, which reduces vigilance in information seeking and processing. Most research originates from the human factors literature, where the prevailing view is that AB occurs only in multitasking environments. Objectives: This review seeks to compare the human factors and health care literature, focusing on the apparent association of AB with multitasking and task complexity. Data sources: EMBASE, Medline, Compendex, Inspec, IEEE Xplore, Scopus, Web of Science, PsycINFO, and Business Source Premiere from 1983 to 2015. Study selection: Evaluation studies where task execution was assisted by automation and resulted in errors were included. Participants needed to be able to verify automation correctness and perform the task manually. Methods: Tasks were identified and grouped. Task and automation type and presence of multitasking were noted. Each task was rated for its verification complexity. Results: Of 890 papers identified, 40 met the inclusion criteria; 6 were in health care. Contrary to the prevailing human factors view, AB was found in single tasks, typically involving diagnosis rather than monitoring, and with high verification complexity. Limitations: The literature is fragmented, with large discrepancies in how AB is reported. Few studies reported the statistical significance of AB compared to a control condition. Conclusion: AB appears to be associated with the degree of cognitive load experienced in decision tasks, and appears to not be uniquely associated with multitasking. Strategies to minimize AB might focus on cognitive load reduction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To the best of our knowledge, existing reviews have focused on AB in relation to traditional computer-based aids rather than AI-driven systems (see Goddard etal. 2012; Lyell and Coiera 2017).”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lyell and Coiera 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Therefore, cognitive overload and trust remain the most crucial factors since, paradoxically, higher automation accuracy can further reinforce user trust, leading to an increased likelihood of AB.”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lyell and Coiera 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the harder the verification complexity, that is the complexity of verifying that the automation is performing correctly, the greater the likelihood of undue trust in the system (Lyell and Coiera 2017).”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lyell and Coiera 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cognitive overload is considered the most significant factor inhibiting users from adequately verifying automation accuracy, thereby increasing AB:”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lyell and Coiera 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This finding aligns with Lyell and Coiera's (2017) consideration that accountability alone is an ineffective intervention.”
From Exploring automation bias in human–AI collaboration: a review and implications for explainable AI (2026) · cites this paper as Lyell and Coiera's (2017)
Experimental — accuracy not yet measured
11 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Sweller, J., Ayres, P., & Kalyuga, S. (2011) · doi:10.1007/978-1-4419-8126-4 · Cited by 5,531
Sweller, J. (2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_446 · Cited by 4,405
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1×
“Tracking task (36) 80 students 8 * 8 mins P< .”
The benefits of imperfect diagnostic automation: a synthesis of the literature
Wickens, C. D., & Dixon, S. R. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500370105 · Cited by 423
Automation Bias: Decision Making and Performance in High-Tech Cockpits
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0801_3 · Cited by 353
Accountability and automation bias
SKITKA, L. J., MOSIER, K., & BURDICK, M. D. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0349 · Cited by 324
Dixon, S. R., & Wickens, C. D. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006778606822 · Cited by 280
Bahner, J. E., Hüper, A.-D., & Manzey, D. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.06.001 · Cited by 279
Effects of Imperfect Automation on Decision Making in a Simulated Command and Control Task
Rovira, E., McGarry, K., & Parasuraman, R. (2007) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007779598082 · Cited by 279
Manzey, D., Reichenbach, J., & Onnasch, L. (2012) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343411433844 · Cited by 277
Wolfstadt, J. I., Gurwitz, J. H., Field, T. S., Lee, M., Kalkar, S., Wu, W., & Rochon, P. A. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0504-5 · Cited by 235
Bailey, N. R., & Scerbo, M. W. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500535301 · Cited by 224
Metzger, U., & Parasuraman, R. (2005) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720053653802 · Cited by 217
McGuirl, J. M., & Sarter, N. B. (2006) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166334 · Cited by 212
Sarter, N. B., & Schroeder, B. (2001) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872001775870403 · Cited by 194
Misdiagnosis of atrial fibrillation and its clinical consequences
Bogun, F., Anh, D., Kalahasty, G., Wissner, E., Bou Serhal, C., Bazzi, R., Douglas Weaver, W., & Schuger, C. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.06.024 · Cited by 165
Automation bias: Empirical results assessing influencing factors
Goddard, K., Roudsari, A., & Wyatt, J. C. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.01.001 · Cited by 163
Computer Decision Support as a Source of Interpretation Error: The Case of Electrocardiograms
Tsai, T. L., Fridsma, D. B., & Gatti, G. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1279 · Cited by 161
NASA aviation safety reporting system
Billings, C. E., Lauber, J. K., Funkhouser, H., Lyman, E., & Huff, E. M. (1976) · NASA Technical Reports Server (NASA) · Cited by 158
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127
Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography
Alberdi, E., Povyakalo, A., Strigini, L., & Ayton, P. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.05.012 · Cited by 124
Coiera, E. W. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-000783 · Cited by 105
Self-citationComparison of a Brain-Based Adaptive System and a Manual Adaptable System for Invoking Automation
Bailey, N. R., Scerbo, M. W., Freeman, F. G., Mikulka, P. J., & Scott, L. A. (2006) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872006779166280 · Cited by 96
The safety and quality of decision support systems.
Coiera, E. W., Westbrook, J. I., & Wyatt, J. C. (2006) · PubMed · Cited by 93
Self-citationPovyakalo, A. A., Alberdi, E., Strigini, L., & Ayton, P. (2013) · Medical Decision Making · doi:10.1177/0272989x12465490 · Cited by 79
Rice, S. C., & McCarley, J. S. (2011) · Journal of Experimental Psychology Applied · doi:10.1037/a0024243 · Cited by 69
Aircrews and automation bias: the advantages of teamwork?
Mosier, K. L., Skitka, L. J., Dunbar, M., & McDonnell, L. (2001) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1101_1 · Cited by 62 · cited 1×
“05 mini-Advanced Concepts Flight Simulator (ACFS) Multi Datalink clearances M I (38) 48 commercial pilots 3 part flights Not reported Not reported”
BackgroundWhich clinical decisions benefit from automation? A task complexity approach
Sintchenko, V., & Coiera, E. W. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00040-6 · Cited by 61
Self-citationTechnology, cognition and error
Coiera, E. W. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003484 · Cited by 55
Self-citationHillson, S. D., Connelly, D. P., & Liu, Y. (1995) · Medical Decision Making · doi:10.1177/0272989x9501500202 · Cited by 49
Stages and Levels of Automation in Support of Space Teleoperations
Li, H., Wickens, C. D., Sarter, N. B., & Sebok, A. L. (2014) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814522830 · Cited by 49
Human Performance Consequences of Automated Decision Aids in States of Sleep Loss
Reichenbach, J., Onnasch, L., & Manzey, D. (2011) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811418222 · Cited by 40
Marx, C., Malich, A., Facius, M., Grebenstein, U., Sauner, D., Pfleiderer, S. O. R., & Kaiser, W. A. (2003) · European Journal of Radiology · doi:10.1016/s0720-048x(03)00144-x · Cited by 39
Bahner, J. E., Elepfandt, M. F., & Manzey, D. (2008) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120805201906 · Cited by 36
Management by Consent in Human-Machine Systems: When and Why It Breaks Down
Olson, W. A., & Sarter, N. B. (2001) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872001775900904 · Cited by 34
Xu, X., Wickens, C. D., & Rantanen, E. M. (2006) · Ergonomics · doi:10.1080/00140130601002658 · Cited by 32
Parasuraman, R., de Visser, E. J., Lin, M., & Greenwood, P. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039675 · Cited by 31
The Effect of Erroneous Computer Interpretation of ECGs on Resident Decision Making
Southern, W. N., & Arnsten, J. H. (2009) · Medical Decision Making · doi:10.1177/0272989x09333125 · Cited by 28
The Duration of Automation Bias in a Realistic Setting
de Boer, R. D., Heems, W. J. H., & Hurts, K. (2014) · International Journal of Aviation Psychology · doi:10.1080/10508414.2014.949205 · Cited by 15
Reichenbach, J., Onnasch, L., & Manzey, D. (2010) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193121005400422 · Cited by 14
Human Factors Perspective on the Reliability of NDT in Nuclear Applications
Bertovic, M., Fahlbruch, B., & Mueller, C. (2013) · Materials Testing · doi:10.3139/120.110431 · Cited by 9
Automation bias and prescribing decision support – rates, mediators and mitigators
Goddard, K. (2012) · City Research Online (City University London) · Cited by 9
Effects of Training and Automation Reliability on Monitoring Performance in a Flight Simulation Task
Singh, I. L., Sharma, H. O., & Parasuraman, R. (2000) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120004401314 · Cited by 9
Monitoring Automation Failures: Effects of Automation Reliability and Task Complexity
Molloy, R., & Parasuraman, R. (1992) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129203601833 · Cited by 6
Monitoring Performance and Mental Workload in an Automated System
Singh, I. L., Singh, A. L., & Saha, P. K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73331-7_47 · Cited by 4
Golchin, K., & Roudsari, A. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-709-3-13 · Cited by 4
The Effect of Different Styles of Human-Machine Interaction on the Nature of Operator Mental Models
Moray, N. P., & Butler, C. (2000) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120004400115 · Cited by 2
Research works printed but not linked: 7
- Mosier, 1996, Human decision makers and automated decision aids: made for each other, 201
- Parasuraman, Performance consequences of automation-induced “complacency.” Int J Aviation Psychol, 1993, 3, 1
- Prinzel, 2005, Individual differences in complacency and monitoring for automation failures, Individual Differences Res, 3, 27
- Singh, 2009, Effects of automation reliability and training on automation-induced complacency and perceived mental workload, J Indian Acad Appl Psychol, 35, 9
- Bagheri, 2004, Considering Subjective Trust and Monitoring Behavior in Assessing Automation-induced “Complacency.”
- Singh, 2010, Performance feedback, mental workload and monitoring efficiency, J Indian Acad Appl Psychol, 36, 151
- Singh, 2001, Effects of manual training and automation reliability on automation induced complacency in flight simulation task, Psychol Stud, 46, 21
Cited by
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341
“African American patients might be harmed if clinical teams started relying on alerts to identify at-risk patients without realizing that the prediction system underdetects patients in that group (automation bias) (16).”
BackgroundEvaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
“Importantly, studies as to the impact of automation bias 53,54 and human– AI interaction biases could also be explored in such a context 55–57 .”
Accountable Artificial Intelligence: Holding Algorithms to Account
Busuioc, M. (2021) · Public Administration Review · doi:10.1111/puar.13293 · Cited by 652
“…algorithmic outputs could conceivably lead to unjustified deference of the decision-maker to algorithm predictions: “Automation bias” (Cummings 2006; Lyell and Coiera 2017; Skitka, Mosier, and Burdick 1999) is a well-documented phenomenon reported in psychological studies manifest in human…”
Background
Metadata
- Authors
- Lyell, D., & Coiera, E.
- Affiliations
- Macquarie University (AU)
- Corresponding author
- Lyell, David
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-08-11
- Volume
- 24
- Issue
- 2
- Pages
- 423-431
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7651899
- OpenAlex
- W2512902140
- Cited by
- 451
- References
- 56
- Citations by year
- 2026: 193 · 2025: 60 · 2024: 55 · 2023: 41 · 2022: 29 · 2021: 27 · 2020: 21 · 2019: 13 · 2018: 7 · 2017: 5
- FWCI
- 1.34
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- HCF Research Foundation
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- automation bias, decision support system, verification complexity, multitasking, cognitive load, task complexity, over-reliance on automation, sustained attention, diagnostic errors, healthcare decision-making, human factors
- MeSH terms
- Attitude to Computers, Automation, Humans, Bias, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.