The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
The ‘problem ’ with automation: inappropriate feedback and interaction, not ‘over-automation’
Open accessAbstract
As automation increasingly takes its place in industry, especially high risk industry, it is often blamed for causing harm and increasing the chance of human error when failures do occur. I propose that the problem is not the presence of automation, but rather its inappropriate design. The problem is that the operations under normal operating conditions are performed appropriately, but there is inadequate feedback and interaction with the humans who must control the overall conduct of the task. When the situations exceed the capabilities of the automatic equipment, then the inadequate feedback leads to difficulties for the human controllers. The problem, I suggest, is that the automation is at an intermediate level of intelligence, powerful enough to take over control that used to be done by people, but not powerful enough to handle all abnormalities. Moreover, its level of intelligence is insufficient to provide the continual, appropriate feedback that occurs naturally among human operators. This is the source of the current difficulties. To solve this problem, the automation should either be made less intelligent or more so, but the current level is quite inappropriate. The overall message is that it is possible to reduce error through appropriate design considerations. Appropriate design should assume the existence of error, it should continually provide feedback, it should continually interact with operators in an effective manner, and it should allow for the worst situations possible. What is needed is a soft, compliant technology, not a rigid, formal one.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
The Psychology of Everyday Things
Massaro, D. W., & Norman, D. A. (1990) · The American Journal of Psychology · doi:10.2307/1423268 · Cited by 3,466
Self-citationIn The Age of the Smart Machine: The Future of Work and Power.
Sirianni, C., & Zuboff, S. (1989) · Contemporary Sociology · doi:10.2307/2073307 · Cited by 3,257
Flight-deck automation: promises and problems
WIENER, E. L., & CURRY, R. E. (1980) · Ergonomics · doi:10.1080/00140138008924809 · Cited by 599
Human Detection and Diagnosis of System Failures
(1981) · Springer eBooks · doi:10.1007/978-1-4615-9230-3 · Cited by 439
The Technology of Team Navigation
(2014) · Intellectual Teamwork · doi:10.4324/9781315807645-16 · Cited by 401
Bainbridge, L. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62897-0 · Cited by 321
Bainbridge, L. (1983) · Analysis, Design and Evaluation of Man–Machine Systems · doi:10.1016/b978-0-08-029348-6.50026-9 · Cited by 219
Sorkin, R. D., Kantowitz, B. H., & Kantowitz, S. C. (1988) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872088803000406 · Cited by 187
Auditory warning sounds in the work environment
Patterson, R. D. (1990) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1990.0091 · Cited by 169
Cited by
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554
Engineering a Safer World: Systems Thinking Applied to Safety
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6305 · Cited by 1,566
Automation, Algorithms, and Beyond: Why Work Design Matters More Than Ever in a Digital World
Parker, S. K., & Grote, G. (2022) · Applied Psychology · doi:10.1111/apps.12241 · Cited by 960
How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control
Sarter, N. B., & Woods, D. D. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049516 · Cited by 849
Kaber, D. B., & Endsley, M. R. (2004) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922021000054335 · Cited by 789
de Winter, J. C. F., Happee, R., Martens, M. H., & Stanton, N. A. (2014) · Transportation Research Part F: Traffic Psychology and Behaviour · doi:10.1016/j.trf.2014.06.016 · Cited by 739
Ten challenges for making automation a ‘team player’ in joint human-agent activity
Klein, G., Woods, D. D., Bradshaw, J. M., Hoffman, R. R., & Feltovich, P. J. (2004) · IEEE Intelligent Systems · doi:10.1109/mis.2004.74 · Cited by 590
Haptic shared control: smoothly shifting control authority?
Abbink, D. A., Mulder, M., & Boer, E. R. (2012) · Cognition, Technology & Work · doi:10.1007/s10111-011-0192-5 · Cited by 525
Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., & Nass, C. (2015) · International Journal on Interactive Design and Manufacturing (IJIDeM) · doi:10.1007/s12008-014-0227-2 · Cited by 497
Sheridan, T. B., & Parasuraman, R. (2005) · Reviews of Human Factors and Ergonomics · doi:10.1518/155723405783703082 · Cited by 474
Situational awareness and safety
Stanton, N. A., Chambers, P. R. G., & Piggott, J. (2001) · Safety Science · doi:10.1016/s0925-7535(01)00010-8 · Cited by 439
Young, M. S., & Stanton, N. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024497709 · Cited by 404
Patterson, E. S., Cook, R. I., & Render, M. L. (2002) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1061 · Cited by 370
Augmenting the algorithm: Emerging human-in-the-loop work configurations
Grønsund, T., & Aanestad, M. (2020) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2020.101614 · Cited by 364
1. How to make automated systems team players
CHRISTOFFERSEN, K., & WOODS, D. (2002) · Advances in Human Performance and Cognitive Engineering Research · doi:10.1016/s1479-3601(02)02003-9 · Cited by 352
Adaptive automation, trust, and self-confidence in fault management of time-critical tasks.
Moray, N., Inagaki, T., & Itoh, M. (2000) · Journal of Experimental Psychology: Applied · doi:10.1037//1076-898x.6.1.44 · Cited by 345
Jamson, A. H., Merat, N., Carsten, O. M. J., & Lai, F. C. H. (2013) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2013.02.008 · Cited by 343
Is Working Memory Working? The Fifteenth Bartlett Lecture
Baddeley, A. (1992) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749208401281 · Cited by 342
Common Ground and Coordination in Joint Activity
Klein, G., Feltovich, P. J., Bradshaw, J. M., & Woods, D. D. (2005) · Organizational Simulation · doi:10.1002/0471739448.ch6 · Cited by 331
MABA-MABA or Abracadabra? Progress on Human-Automation Co-ordination
Dekker, S. W. A., & Woods, D. D. (2002) · Cognition, Technology & Work · doi:10.1007/s101110200022 · Cited by 328
Activities of the rescue robots at the World Trade Center from 11-21 september 2001
Murphy, R. R. (2004) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2004.1337826 · Cited by 325
A Topology of Shared Control Systems—Finding Common Ground in Diversity
Abbink, D. A., Carlson, T., Mulder, M., de Winter, J. C. F., Aminravan, F., Gibo, T. L., & Boer, E. R. (2018) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2018.2791570 · Cited by 297
Broadhurst, K., Wastell, D., White, S., Hall, C., Peckover, S., Thompson, K., Pithouse, A., & Davey, D. (2010) · British Journal of Social Work · doi:10.1093/bjsw/bcn162 · Cited by 289
The “Out-of-the-Loop” concept in automated driving: proposed definition, measures and implications
Merat, N., Seppelt, B., Louw, T., Engström, J., Lee, J. D., Johansson, E., Green, C. A., Katazaki, S., Monk, C., Itoh, M., et al. (2019) · Cognition, Technology & Work · doi:10.1007/s10111-018-0525-8 · Cited by 282
Automation and situation awareness.
Endsley, M. R. (2018) · doi:10.1201/9781315137957-8 · Cited by 282
Sarter, N. B., & Woods, D. D. (1994) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0401_1 · Cited by 280
Pilot Interaction With Cockpit Automation: Operational Experiences With the Flight Management System
Sarter, N. B., & Woods, D. D. (1992) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0204_5 · Cited by 255
Automation technology and sense of control: A window on human agency
Berberian, B., Sarrazin, J.-C., Le Blaye, P., & Haggard, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0034075 · Cited by 190
Kunze, A., Summerskill, S. J., Marshall, R., & Filtness, A. J. (2019) · Ergonomics · doi:10.1080/00140139.2018.1547842 · Cited by 165
Hamacher, A., Bianchi-Berthouze, N., Pipe, A. G., & Eder, K. (2016) · 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/roman.2016.7745163 · Cited by 135
Overtrust in External Cues of Automated Vehicles: An Experimental Investigation
Holländer, K., Wintersberger, P., & Butz, A. (2019) · Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/3342197.3344528 · Cited by 109
Anthropomorphizing information to enhance trust in autonomous vehicles
Niu, D., Terken, J., & Eggen, B. (2018) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20745 · Cited by 90
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
Auditory interfaces in automated driving: an international survey
Bazilinskyy, P., & de Winter, J. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.13 · Cited by 83
Effective human–AI work design for collaborative decision-making
Jain, R., Garg, N., & Khera, S. N. (2023) · Kybernetes · doi:10.1108/k-04-2022-0548 · Cited by 85
Understanding human management of automation errors
McBride, S. E., Rogers, W. A., & Fisk, A. D. (2014) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2013.817625 · Cited by 82
Nezami, F. N., Wächter, M. A., Maleki, N., Spaniol, P., Kühne, L. M., Haas, A., Pingel, J. M., Tiemann, L., Nienhaus, F., Keller, L., et al. (2021) · Sensors · doi:10.3390/s21051879 · Cited by 16
Supporting operator reliance on automation through continuous feedback
Seppelt, B. D. (2018) · doi:10.17077/etd.90t9uqnh · Cited by 11
Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · arXiv · doi:10.48550/arxiv.2402.11364 · Cited by 7
Metadata
- Authors
- Norman, D. A.
- Venue
- Philosophical Transactions of the Royal Society of London. B, Biological Sciences
- Publisher
- The Royal Society
- Published
- 1990-04-12
- Volume
- 327
- Issue
- 1241
- Pages
- 585-593
- ISSN
- 0080-4622 (ISSN-L)
- 2054-0280
- OpenAlex
- W2153246402
- Cited by
- 510
- References
- 12
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.