The DOI matches its Crossref record; 64 of the 67 references it cites were found in a registry or an index.
Algorithmic Decision-Making and the Control Problem
Open accessAbstract
The danger of human operators devolving responsibility to machines and failing to detect cases where they fail has been recognised for many years by industrial psychologists and engineers studying the human operators of complex machines. We call it “the control problem”, understood as the tendency of the human within a human–machine control loop to become complacent, over-reliant or unduly diffident when faced with the outputs of a reliable autonomous system. While the control problem has been investigated for some time, up to this point its manifestation in machine learning contexts has not received serious attention. This paper aims to fill that gap. We argue that, except in certain special circumstances, algorithmic decision tools should not be used in high-stakes or safety-critical decisions unless the systems concerned are significantly “better than human” in the relevant domain or subdomain of decision-making. More concretely, we recommend three strategies to address the control problem, the most promising of which involves a complementary (and potentially dynamic ) coupling between highly proficient algorithmic tools and human agents working alongside one another. We also identify six key principles which all such human–machine systems should reflect in their design. These can serve as a framework both for assessing the viability of any such human–machine system as well as guiding the design and implementation of such systems generally.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Human oversight has also been subject to criticism, with concerns being raised about the efficacy of oversight in practice and its potential role as a mere vehicle to legitimize imperfect AI systems [31, 89].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This way, the design of the system can support the attention of decision makers by varying reliability over time (Goddard et al., 2012; Zerilli et al., 2019).”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Alternatively, oversight persons may only receive the AI-based output after they have made up their own mind about a specific case because then they would not initially be affected by the AIbased output and may be in a better position to judge whether the AI-based output is appropriate [31, 89].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]
The cited source says what the claim attributes to it.
In the citing paper
“Diffusion of responsibility also occurs in tasks humans share with automatic systems”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Since automation bias tends to occur especially when the system is perceived as highly reliable (Zerilli et al., 2019) and has been working error-free for a long time, i.e., has low failure rates (Parasuraman and Manzey, 2010), companies should make decision makers aware of possible system failures not once, but on a regular base.”
From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019
In the citing paper
“One of the consequences can be the shift from operator to supervisor, that is, expert work becomes more about monitoring and surveillance than before (Zerilli et al. 2019).”
From Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care (2022) · cites this paper as Zerilli et al. 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Quarterly Journal of Economics
Kuznets, S. (2024) · Routledge eBooks · doi:10.4324/9781003549741-8 · Cited by 7,177
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063712 · Cited by 6,886
Newell, A. (1972) · Cited by 6,222
The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L. Whitwell, J., Ward, C., et al. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21049 · Cited by 4,347
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2018) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,320
Bainbridge, L. (1983) · Automatica · doi:10.1016/0005-1098(83)90046-8 · Cited by 2,205
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710
Extraneous factors in judicial decisions
Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018033108 · Cited by 1,431
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
(2018) · Cited by 1,167
Research works printed but not linked: 8
- AI Now. (2018). Litigating algorithms: Challenging government use of algorithmic decision systems. New York: AI Now Institute.
- Amoroso, N., La Rocca, M., Bruno, S., Maggipinto, T., Monaco, A., Bellotti, R., Tangaro, S., the Alzheimer’s Disease Neuroimaging Initiative. (2017). Brain structural connectivity atrophy in Alzheimer’s disease. arXiv:1709.02369v1. · arXiv
- Bagheri, N., & Jamieson, G. A. (2004). Considering subjective trust and monitoring behavior in assessing automation-induced “complacency”. In D. A. Vicenzi, M. Mouloua, & O. A. Hancock (Eds.), Human performance, situation awareness, and automation: Current research and trends (pp. 54–59). Mahwah, NJ: Erlbaum.
- Blomberg, T., Bales, W., Mann, K., Meldrum, R., Nedelec, J. (2010). Validation of the COMPAS risk assessment classification instrument. Center for Criminology and Public Policy Research College of Criminology and Criminal Justice Florida State University. https://arxiv.org/pdf/1311.2901.pdf. · arXiv
- Cebon, D. (2015). Responses to autonomous vehicles. Ingenia,62, 10.
- Fitts, P. M. (1951). Human engineering for an effective air navigation and traffic control system. Washington D.C.: National Research Council.
- IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2017). Ethically aligned design (version 2). https://ethicsinaction.ieee.org.
- Stanton, N. A. (2015). Responses to autonomous vehicles. Ingenia,62, 9.
Cited by
Sorted by citations
Items 101–144 of 144
Dias, K. C., Lourenço, A., Silva, G. V., Cruz, M. V. M., & Barbalho, S. C. M. (2022) · International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) · doi:10.1109/iceccme55909.2022.9988362 · Cited by 1
(2022) · Cambridge University Press eBooks · doi:10.1017/9781009071215.009
Are We Obligated to Employ Algorithmic Decision-Making?
Pfister, J. (2022) · Springer eBooks · doi:10.1007/978-3-030-90513-2_7
Muhić, H., Huskić, S., & Hibert, M. (2023) · Pregled: časopis za društvena pitanja / Periodical for social issues · doi:10.48052/19865244.2023.1.231
Design and Monitoring Automatization Of Exhaust Fan Through Smartphone at PT Semen Tonasa
Asyik, P., & Nugroho, E. H. (2023) · Inspiration Jurnal Teknologi Informasi dan Komunikasi · doi:10.35585/inspir.v13i1.52
Human Interaction in Machine Learning (ML) for Healthcare
Ibrahim, S. A. H., & Dehghani, F. (2023) · Springer eBooks · doi:10.1007/978-3-319-08234-9_522-1
Human Interaction in Machine Learning (ML) for Healthcare
Ibrahim, S. A. H., & Dehghani, F. (2024) · Springer eBooks · doi:10.1007/978-3-031-23161-2_522
Kaklauskiene, L., & Kaklauskas, A. (2024) · Studies in Informatics and Control · doi:10.24846/v33i1y202404
Andrade, F., & Rebelo, D. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4882806
The Dawn of the Algorithmic Researcher
(2024) · Advances in knowledge acquisition, transfer, and management book series/Advances in knowledge acquisition, transfer and management book series · doi:10.4018/979-8-3693-6905-0.ch001
Anthropocentric AI for EU Consumer Lending
Rebelo, D. M., Andrade, F., & Nováis, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73497-7_25
Navigating ethical horizon: A dive into metaverse and analytics of things in healthcare system
Jasrotia, S. S., Kamila, M. K., & Kushwaha, P. S. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-24049-2.00001-7
Ibrahim, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5251305
Navigating the Tension Between Algorithmic and Human Ethics: Toward a Responsible Balance
Arencibia, M. G., Ordóñez, H., & González-Sanabria, J.-S. (2025) · Revista Facultad de Ingeniería · doi:10.19053/uptc.01211129.v34.n71.2025.19650
Hoke, T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-91235-1_7
Von der Entwicklung zur Praxis: KI-gestützte Systeme in der Sozialen Arbeit einführen
Plafky, C. S., & Badertscher, H. (2025) · Springer eBooks · doi:10.1007/978-3-658-49009-6_9
KI-Anwendungen in der Praxis Soziale Arbeit und angrenzender Professionen – ein Überblick
Plafky, C. S., & Badertscher, H. (2025) · Springer eBooks · doi:10.1007/978-3-658-49009-6_2
Prädiktive Analytik und Entscheidungs unterstützung in der Praxis Sozialer Arbeit
Plafky, C. S., & Badertscher, H. (2025) · Springer eBooks · doi:10.1007/978-3-658-49009-6_4
Plafky, C. S., & Badertscher, H. (2025) · Springer eBooks · doi:10.1007/978-3-658-49009-6_8
A study on AI-enhanced evaluation of store attributes affecting customer loyalty in Hyderabad city
Babu, N. S., Ganesh, C., Madhavi, B., Raju, N. G., Adulapuram, D., & Hasan, M. M. R. (2025) · AIP conference proceedings · doi:10.1063/5.0306006
Divine Determinism to Digital Determinism: The Paradigm Shift of Fate in the 21st Century
Adela, F. P., & Bangsawan, T. A. (2025) · Pharos Journal of Theology · doi:10.46222/pharosjot.107.27 · Cited by 1
Plafky, C. S., & Badertscher, H. (2025) · Springer eBooks · doi:10.1007/978-3-658-49009-6_1
Artificial Intelligence-Induced Algorithmic Bias in Public Services
ÖZER, G. S. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1484-6.ch007
AI-based Consumers’ Creditworthiness Fate Needs EU ‘Lawgorithmics’ to Innovate
Rebelo, D. M., Andrade, F., & Nováis, P. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769233
Relevance of Spirituality to AI Ethics in BANI Times
Kanuri, R. R. (2026) · Responsible leadership and sustainable management · doi:10.1007/978-981-95-5929-9_13
Claudius, B. H., Setyo, I. C., Fernando, J., & Setiawan, S. (2025) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss66954.2025.11389314
Accuracy is not all you need! The Reasons to Require AI Explainability
Buijsman, S. (2026) · Minds and Machines · doi:10.1007/s11023-026-09768-x · Cited by 1
Mihai, R. G., Bergh, L. I. V., Cayeux, E., Daireaux, B., & Lootz, E. (2026) · SPE/IADC Drilling Conference · doi:10.2118/230731-ms
Public Perspectives on Robot Intervention: Insights from a Low-Supervision Decision Exhibit
Mital, A., Gnisa, F., Norman, U., & Weinberger, N. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794569
Vladika, J., Fichtl, A. M., & Matthes, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-45245-2
Between Negligence and Malfunction: How to Address Responsibility for AI Failures
Boutin, B. (2026) · T.M.C. Asser Press eBooks · doi:10.1007/978-94-6265-759-5_13
Bertrand, A. (2024) · theses.fr (ABES) · doi:10.70675/216d3647z78bcz4c68z9ba9z2d0a029cf1b4 · Cited by 1
Digital Marketing Strategy of Clothing Brands Based on Big Data
Li, H., & Li, S. (2026) · Journal of Cases on Information Technology · doi:10.4018/jcit.407185
Liu, Q. (2026) · International Journal of Digital Law and Governance · doi:10.1515/ijdlg-2025-0021
Developing an Assessment Tool for AI Guidelines
Pelloth, J., & Zweig, K. A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812345
Dizdar, E. B., Polat, S., & Himmetoğlu, B. (2026) · Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi · doi:10.17240/aibuefd.2026..-1831922
From product to operative artefact: practical authority and operational answerability in AI ethics
Velotto, G. (2026) · AI and Ethics · doi:10.1007/s43681-026-01285-y
From Journalistic Gatekeeping to Algorithmic Governance
Shin, D. (2026) · Oxford University Press eBooks · doi:10.1093/9780197849644.003.0002
Mahilum, T., Jr., Go, M., Magno, R., Mancenido, A., & Tumaroy, L. A. (2026) · JPAIR Multidisciplinary Research · doi:10.7719/jpair.v66i1.1065
Carpenter, R. E., Dial, B., Huyler, D., Krouse, A. R., & Sun, J. (2026) · Administrative Sciences · doi:10.3390/admsci16100467
Sharma, R. (2026) · Employee Relations · doi:10.1108/er-10-2025-0901
Biswas, M. I., Talukder, M. S., & Opel, M. I. T. (2026) · Research Square · doi:10.21203/rs.3.rs-10395816/v1
Ma, J., Chen, T., Zuo, R., Johnson, T. P., Welch, E. W., Frandell, A., Arroyo‐Machado, W., & Michalegko, L. (2026) · Policy & Internet · doi:10.1002/poi3.70064
Metadata
- Authors
- Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C.
- Affiliations
- University of Cambridge (GB)
- University of Otago (NZ)
- Corresponding author
- Zerilli, John
- Venue
- Minds and Machines
- Published
- 2019-12
- Volume
- 29
- Issue
- 4
- Pages
- 555-578
- ISSN
- 0924-6495 (ISSN-L)
- 1572-8641
- OpenAlex
- W2996655830
- Cited by
- 145
- References
- 67
- Citations by year
- 2026: 25 · 2025: 30 · 2024: 29 · 2023: 18 · 2022: 20 · 2021: 15 · 2020: 7 · 2019: 1
- FWCI
- 13.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Minds and Machines (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- New Zealand Law Foundation (2016/ILP/10)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- control problems, algorithmic decision-making, human-machine systems, high-stakes decision making, human-machine collaboration, human-machine interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
