Automation bias in intelligent time critical decision support systems
Abstract
Various levels of automation can be introduced by intelligent decision support systems, from fully automated, where the operator is completely left out of the decision process, to minimal levels of automation, where the automation only makes recommendations and the operator has the final say. For rigid tasks that require no flexibility in decision-making and with a low probability of system failure, higher levels of automation often provide the best solution. However, in time critical environments with many external and changing constraints such as air traffic control and military command and control operations, higher levels of automation are not advisable because of the risks and the complexity of both the system and the inability of the automated decision aid to be perfectly reliable. Human-in-the-loop designs, which employ automation for redundant, manual, and monotonous tasks and allow operators active participation, provide not only safety benefits, but also allow a human operator and a system to respond more flexibly to uncertain and unexpected events. However, there can be measurable costs to human performance when automation is used, such as loss of situational awareness, complacency, skill degradation, and automation bias. This paper will discuss the influence of automation bias in intelligent decision support systems, particularly those in aviation domains. Automation bias occurs in decision-making because humans have a tendency to disregard or not search for contr…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automation bias often occurs in a multitasking setting and under a high workload, as it serves as a decision-making heuristic, which saves time and cognitive effort.
“Automation bias often occurs in a multitasking setting and under a high workload, as it serves as a decision-making heuristic, which saves time and cognitive effort (Parasuraman and Manzey, 2010; Cummings, 2017).”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Cummings, 2017
Cites
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · see citing works
ENDSLEY, M. R., & KABER, D. B. (1999) · Ergonomics · doi:10.1080/001401399185595 · Cited by 1,088 · see citing works
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · see citing works
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · see citing works
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384 · see citing works
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · see citing works
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217 · see citing works
<p>Automation and Accountability in Decision Support System Interface Design</p>
Cummings, M. L. (2006) · Journal of Technology Studies · doi:10.21061/jots.v32i1.a.4 · Cited by 155 · see citing works
Self-citationWhen speed kills: Lethal autonomous weapon systems, deterrence and stability
Horowitz, M. C. (2019) · Journal of Strategic Studies · doi:10.1080/01402390.2019.1621174 · Cited by 148 · see citing works
Algorithmic Decision-Making and the Control Problem
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Minds and Machines · doi:10.1007/s11023-019-09513-7 · Cited by 145 · see citing works
The Placebo Effect of Artificial Intelligence in Human–Computer Interaction
Kosch, T., Welsch, R., Chuang, L., & Schmidt, A. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3529225 · Cited by 137 · see citing works
Kroll, J. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0084 · Cited by 132 · see citing works
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119 · see citing works
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · see citing works
Human control of AI systems: from supervision to teaming
Tsamados, A., Floridi, L., & Taddeo, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00489-4 · Cited by 92 · see citing works
Datafied child welfare services: unpacking politics, economics and power
Redden, J., Dencik, L., & Warne, H. (2020) · Policy Studies · doi:10.1080/01442872.2020.1724928 · Cited by 89 · see citing works
Automated Decision-Making and Big Data: Concerns for People With Mental Illness
Monteith, S., & Glenn, T. (2016) · Current Psychiatry Reports · doi:10.1007/s11920-016-0746-6 · Cited by 62 · see citing works
Check the box! How to deal with automation bias in AI-based personnel selection
Kupfer, C., Prassl, R., Fleiß, J., Malin, C., Thalmann, S., & Kubicek, B. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1118723 · Cited by 59 · see citing works
Towards Unbiased and Accurate Deferral to Multiple Experts
Keswani, V., Lease, M., & Kenthapadi, K. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462516 · Cited by 45 · see citing works
Geburczyk, F. (2021) · Computer Law & Security Review · doi:10.1016/j.clsr.2021.105538 · Cited by 7 · see citing works
- Authors
- Cummings, M. L.
- Affiliations
- Massachusetts Institute of Technology (US)
- Corresponding author
- Cummings, M.L.
- ORCID
- Cummings, M.L.
- Venue
- Decision Making in Aviation
- Publisher
- Routledge
- Published
- 2017-07-05
- Pages
- 289-294
- OpenAlex
- W2124002192
- Cited by
- 233
- References
- 5
- Citations by year
- 2026: 24 · 2025: 20 · 2024: 24 · 2023: 33 · 2022: 11 · 2021: 15 · 2020: 15 · 2019: 9 · 2018: 11 · 2017: 6 · 2016: 6 · 2015: 4 · 2014: 3 · 2013: 6 · 2012: 4
- FWCI
- 43.22
- Type
- book chapter
- Language
- en
- Open access
- closed
- Locations
- Routledge eBooks (publishedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Engineering › Control and Systems Engineering › Military Strategy and Technology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Risk and Safety Analysis
- Keywords
- automation bias, intelligent decision support system, time-critical decision making, human-in-the-loop, operator performance, situational awareness, complacency, deskilling, air traffic control, military command and control, human cognitive limitations, automated decision aids
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
