The DOI matches its Crossref record; 64 of the 67 references it cites were found in a registry or an index.
Algorithmic Decision-Making and the Control Problem
Open accessAbstract
The danger of human operators devolving responsibility to machines and failing to detect cases where they fail has been recognised for many years by industrial psychologists and engineers studying the human operators of complex machines. We call it “the control problem”, understood as the tendency of the human within a human–machine control loop to become complacent, over-reliant or unduly diffident when faced with the outputs of a reliable autonomous system. While the control problem has been investigated for some time, up to this point its manifestation in machine learning contexts has not received serious attention. This paper aims to fill that gap. We argue that, except in certain special circumstances, algorithmic decision tools should not be used in high-stakes or safety-critical decisions unless the systems concerned are significantly “better than human” in the relevant domain or subdomain of decision-making. More concretely, we recommend three strategies to address the control problem, the most promising of which involves a complementary (and potentially dynamic ) coupling between highly proficient algorithmic tools and human agents working alongside one another. We also identify six key principles which all such human–machine systems should reflect in their design. These can serve as a framework both for assessing the viability of any such human–machine system as well as guiding the design and implementation of such systems generally.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human oversight has also been subject to criticism, with concerns being raised about the efficacy of oversight in practice and its potential role as a mere vehicle to legitimize imperfect AI systems [31, 89].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This way, the design of the system can support the attention of decision makers by varying reliability over time (Goddard et al., 2012; Zerilli et al., 2019).”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Alternatively, oversight persons may only receive the AI-based output after they have made up their own mind about a specific case because then they would not initially be affected by the AIbased output and may be in a better position to judge whether the AI-based output is appropriate [31, 89].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]
The cited source says what the claim attributes to it.
In the citing paper
“Diffusion of responsibility also occurs in tasks humans share with automatic systems”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Since automation bias tends to occur especially when the system is perceived as highly reliable (Zerilli et al., 2019) and has been working error-free for a long time, i.e., has low failure rates (Parasuraman and Manzey, 2010), companies should make decision makers aware of possible system failures not once, but on a regular base.”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
In the citing paper
“One of the consequences can be the shift from operator to supervisor, that is, expert work becomes more about monitoring and surveillance than before (Zerilli et al. 2019).”
From Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care (2022) · cites this paper as Zerilli et al. 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Quarterly Journal of Economics
Kuznets, S. (2024) · Routledge eBooks · doi:10.4324/9781003549741-8 · Cited by 7,177
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063712 · Cited by 6,886
Newell, A. (1972) · Cited by 6,222
The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L. Whitwell, J., Ward, C., et al. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21049 · Cited by 4,347
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2018) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,320
Bainbridge, L. (1983) · Automatica · doi:10.1016/0005-1098(83)90046-8 · Cited by 2,205
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710
Extraneous factors in judicial decisions
Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018033108 · Cited by 1,431
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
(2018) · Cited by 1,167
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles
(2021) · BSI Standards · doi:10.3403/30443080u · Cited by 992
From Here to Autonomy: Lessons Learned From Human–Automation Research
Endsley, M. R. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816681350 · Cited by 898
Αλέτρας, Ν., Tsarapatsanis, D., Preoțiuc-Pietro, D., & Lampos, V. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.93 · Cited by 813
Machine, platform, crowd: Harnessing our digital future
Peregrina, Á. F. A. (2018) · Economía industrial · Cited by 681
Flight-deck automation: promises and problems
WIENER, E. L., & CURRY, R. E. (1980) · Ergonomics · doi:10.1080/00140138008924809 · Cited by 599
Meaningful Human Control over Autonomous Systems: A Philosophical Account
Santoni de Sio, F., & van den Hoven, J. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00015 · Cited by 566
Fildes, R. A., Goodwin, P., Lawrence, M. J., & Nikolopoulos, K. I. (2009) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2008.11.010 · Cited by 447
Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420
Self-citationMental workload : its theory and measurement
Moray, N. P. (1979) · Medical Entomology and Zoology · Cited by 343
Accountability and automation bias
SKITKA, L. J., MOSIER, K., & BURDICK, M. D. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0349 · Cited by 324
Monitoring an Automated System for a Single Failure: Vigilance and Task Complexity Effects
MOLLOY, R., & Parasuraman, R. (1996) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089606380211 · Cited by 321
Bainbridge, L. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62897-0 · Cited by 321
Machine, Platform, Crowd: Harnessing Our Digital Future
Brynjolfsson, E., & McAfee, A. P. (2017) · Cited by 313
Man-Machine Systems: Information, Control and Decision Models of Human Performance
Menon, U., Sheridan, T. B., & Ferrel, W. R. (1976) · Operational Research Quarterly (1970-1977) · doi:10.2307/3009156 · Cited by 272
Automation Bias in Intelligent Time Critical Decision Support Systems
Cummings, M. (2004) · AIAA 1st Intelligent Systems Technical Conference · doi:10.2514/6.2004-6313 · Cited by 269
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244
Is partially automated driving a bad idea? Observations from an on-road study
Banks, V. A., Eriksson, A., O'Donoghue, J., & Stanton, N. A. (2017) · Applied Ergonomics · doi:10.1016/j.apergo.2017.11.010 · Cited by 244
Automation bias in intelligent time critical decision support systems
Cummings, M. L. (2017) · Decision Making in Aviation · doi:10.4324/9781315095080-17 · Cited by 233
The History of Human Factors and Ergonomics
Meister, D. (2018) · CRC Press eBooks · doi:10.1201/9781315276069 · Cited by 232
From fly-by-wire to drive-by-wire: Safety implications of automation in vehicles
Stanton, N. A., & Marsden, P. (1996) · Safety Science · doi:10.1016/s0925-7535(96)00067-7 · Cited by 202
The Oxford Handbook of Law, Regulation and Technology
Novitz, T. A. (2016) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199680832.001.0001 · Cited by 190
Man-machine systems—Information, control, and decision models of human performance
Shneiderman, B. (1975) · Information Processing & Management · doi:10.1016/0306-4573(75)90039-4 · Cited by 186
Driver Vigilance in Automated Vehicles: Hazard Detection Failures Are a Matter of Time
Greenlee, E. T., DeLucia, P. R., & Newton, D. C. (2018) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720818761711 · Cited by 178
Banks, V. A., Plant, K. L., & Stanton, N. A. (2017) · Safety Science · doi:10.1016/j.ssci.2017.12.023 · Cited by 176
Past and future drought in Mongolia
Hessl, A., Anchukaitis, K. J., Jelsema, C., Cook, B. I., Byambasuren, O., Leland, C., Nachin, B., Pederson, N., Tian, H., & Andreu‐Hayles, L. (2018) · Science Advances · doi:10.1126/sciadv.1701832 · Cited by 165
Human-Computer Interaction in the Control of Dynamic Systems
Rouse, W. B. (1981) · ACM Computing Surveys · doi:10.1145/356835.356839 · Cited by 150
Models of human problem solving: Detection, diagnosis, and compensation for system failures
Rouse, W. B. (1983) · Automatica · doi:10.1016/0005-1098(83)90025-0 · Cited by 149
"Manual and automatic control"
Ferrell, W. R. (1969) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1969.1099124 · Cited by 143
Ironies of Automation: Still Unresolved After All These Years
Strauch, B. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2732506 · Cited by 139
Man-Machine Systems (Information, Control, and Decision Models of Human Performance)
Sheridan, T. B., Ferrell, W. R., & Tomizuka, M. (1975) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3426860 · Cited by 137
The Human Operator in Process Control
Edwards, E., & Lees, F. P. (1974) · Medical Entomology and Zoology · Cited by 123
5 Manual and automatic control
Agachi, P. S., Cristea, M. V., & Makhura, E. P. (2020) · De Gruyter eBooks · doi:10.1515/9783110647938-005 · Cited by 110
Distributed situation awareness
Stanton, N. A. (2015) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2015.1106615 · Cited by 94
The Effects of Participatory Mode and Task Workload on the Detection of Dynamic System Failures
Wickens, C. D., & Kessel, C. (1979) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1979.4310070 · Cited by 89
Baxter, G. D., Rooksby, J., Wang, Y., & Khajeh‐Hosseini, A. (2012) · European Conference on Cognitive Ergonomics · doi:10.1145/2448136.2448149 · Cited by 75
Investigation on the impact of human-automation interaction in maritime operations
Pazouki, K., Forbes, N., Norman, R. A., & Woodward, M. D. (2018) · Ocean Engineering · doi:10.1016/j.oceaneng.2018.01.103 · Cited by 71
Damaska, M. R. (2017) · Yale University Press eBooks · doi:10.12987/9780300146479 · Cited by 65
Human Factors in Automotive Engineering and Technology
Walker, G. H. (2017) · CRC Press eBooks · doi:10.1201/9781315587356 · Cited by 52
SNF: synthesizing high performance NFV service chains
Katsikas, G. P., Enguehard, M., Kuźniar, M., Maguire, G. Q., & Kostić, D. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.98 · Cited by 52
Models of Human Problem Solving: Detection, Diagnosis, and Compensation for System Failures
Rouse, W. B. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62903-3 · Cited by 21
Johannsen, G., Rijnsdorp, J. E., Ifip, & Ifors (1983) · Cited by 17
Multiple realization and the commensurability of taxonomies
Zerilli, J. (2017) · Synthese · doi:10.1007/s11229-017-1599-1 · Cited by 17
Self-citationCognitive Elements of Human Decision Making
Pohl, J. G. (2008) · Studies in computational intelligence · doi:10.1007/978-3-540-76829-6_2 · Cited by 16
Man's role in man-machine systems
Margulies, F., & Zemanek, H. (1983) · Automatica · doi:10.1016/0005-1098(83)90031-6 · Cited by 13
Man-Machine Systems - Introduction and Background
Johannsen, G. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62878-7 · Cited by 12
Bygrave, L. A. (2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199680832.013.52 · Cited by 10
Human-computer dialogue design considerations
Williges, R. C., & Williges, B. H. (1983) · Automatica · doi:10.1016/0005-1098(83)90045-6 · Cited by 8
Man's Role in Man-Machine Systems
Margulies, F., & Zemanek, H. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62896-9 · Cited by 7
Human-Computer Dialogue Design Considerations
Williges, R. C., & Williges, B. H. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62912-4 · Cited by 6
Man-Machine Interaction in Computer-Aided Design Systems
Hatvany, J., & Guedj, R. A. (1982) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)62911-2 · Cited by 5
MAN-MACHINE INTERACTION IN COMPUTER-AIDED DESIGN SYSTEMS
Hatvany, J., & Guedj, R. A. (1983) · Elsevier eBooks · doi:10.1016/b978-0-08-029348-6.50040-3 · Cited by 4
Automated vehicles may encourage a new breed of distracted drivers
Cunningham, M. L., & Regan, M. (2018) · doi:10.64628/aa.k5yhh3669 · Cited by 4
Research works printed but not linked: 8
- AI Now. (2018). Litigating algorithms: Challenging government use of algorithmic decision systems. New York: AI Now Institute.
- Amoroso, N., La Rocca, M., Bruno, S., Maggipinto, T., Monaco, A., Bellotti, R., Tangaro, S., the Alzheimer’s Disease Neuroimaging Initiative. (2017). Brain structural connectivity atrophy in Alzheimer’s disease. arXiv:1709.02369v1. · arXiv
- Bagheri, N., & Jamieson, G. A. (2004). Considering subjective trust and monitoring behavior in assessing automation-induced “complacency”. In D. A. Vicenzi, M. Mouloua, & O. A. Hancock (Eds.), Human performance, situation awareness, and automation: Current research and trends (pp. 54–59). Mahwah, NJ: Erlbaum.
- Blomberg, T., Bales, W., Mann, K., Meldrum, R., Nedelec, J. (2010). Validation of the COMPAS risk assessment classification instrument. Center for Criminology and Public Policy Research College of Criminology and Criminal Justice Florida State University. https://arxiv.org/pdf/1311.2901.pdf. · arXiv
- Cebon, D. (2015). Responses to autonomous vehicles. Ingenia,62, 10.
- Fitts, P. M. (1951). Human engineering for an effective air navigation and traffic control system. Washington D.C.: National Research Council.
- IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2017). Ethically aligned design (version 2). https://ethicsinaction.ieee.org.
- Stanton, N. A. (2015). Responses to autonomous vehicles. Ingenia,62, 9.
Cited by
Sorted by citations
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“in a directly positive outcome, and can in fact create the opposite[185, 249].”
BackgroundAlon-Barkat, S., & Busuioc, M. (2023) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muac007 · Cited by 469
Ali, E. S., Hasan, M. K., Hassan, R., Saeed, R. A., Hassan, M. B., Islam, S., Nafi, N. S., & Bevinakoppa, S. (2021) · Security and Communication Networks · doi:10.1155/2021/8868355 · Cited by 257
How transparency modulates trust in artificial intelligence
Zerilli, J., Bhatt, U., & Weller, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100455 · Cited by 257
Self-citationInterpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780367816377-16 · Cited by 276
Metadata
- Authors
- Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C.
- Affiliations
- University of Cambridge (GB)
- University of Otago (NZ)
- Corresponding author
- Zerilli, John
- Venue
- Minds and Machines
- Published
- 2019-12
- Volume
- 29
- Issue
- 4
- Pages
- 555-578
- ISSN
- 0924-6495 (ISSN-L)
- 1572-8641
- OpenAlex
- W2996655830
- Cited by
- 145
- References
- 67
- Citations by year
- 2026: 25 · 2025: 30 · 2024: 29 · 2023: 18 · 2022: 20 · 2021: 15 · 2020: 7 · 2019: 1
- FWCI
- 13.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Minds and Machines (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- New Zealand Law Foundation (2016/ILP/10)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- control problems, algorithmic decision-making, human-machine systems, high-stakes decision making, human-machine collaboration, human-machine interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
