CoChat
Sign Up
Cited by 145Human-Automation Interaction and SafetyEthics and Social Impacts of AISafety Systems Engineering in Autonomy

Algorithmic Decision-Making and the Control Problem

Open access
Zerilli, John, Knott, Alistair, Maclaurin, James, Gavaghan, Colin·2019·Minds and Machines
DOI 10.1007/s11023-019-09513-7
Open this paper in CoChat — free accountOpen PDF

Abstract

The danger of human operators devolving responsibility to machines and failing to detect cases where they fail has been recognised for many years by industrial psychologists and engineers studying the human operators of complex machines. We call it “the control problem”, understood as the tendency of the human within a human–machine control loop to become complacent, over-reliant or unduly diffident when faced with the outputs of a reliable autonomous system. While the control problem has been investigated for some time, up to this point its manifestation in machine learning contexts has not received serious attention. This paper aims to fill that gap. We argue that, except in certain special circumstances, algorithmic decision tools should not be used in high-stakes or safety-critical decisions unless the systems concerned are significantly “better than human” in the relevant domain or subdomain of decision-making. More concretely, we recommend three strategies to address the control problem, the most promising of which involves a complementary (and potentially dynamic ) coupling between highly proficient algorithmic tools and human agents working alongside one another. We also identify six key principles which all such human–machine systems should reflect in their design. These can serve as a framework both for assessing the viability of any such human–machine system as well as guiding the design and implementation of such systems generally.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11023-019-09513-7

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human oversight has been subject to criticism, with concerns about the efficacy of oversight in practice and its potential role as a mere vehicle to legitimize imperfect AI systems.

    “Human oversight has also been subject to criticism, with concerns being raised about the efficacy of oversight in practice and its potential role as a mere vehicle to legitimize imperfect AI systems [31, 89].”

    From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]

    The Introduction and Conclusion describe how human supervisors can become complacent and hand effective control to an algorithm, supporting concern about oversight’s practical efficacy. The passages do not say that human oversight serves to legitimize imperfect AI systems; the discussion of an unvetted system concerns an assumption that the system itself had been validated. Page numbers are not available in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Deliberately programming errors into AI-based decision support systems and pointing them out when they are overlooked can support the attention of decision makers by varying reliability over time.

    “This way, the design of the system can support the attention of decision makers by varying reliability over time (Goddard et al., 2012; Zerilli et al., 2019).”

    From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019

    Section 5.1 (System Reliability; page unavailable) describes catch trials that deliberately generate system errors to keep human monitors attentive, and says they may engage people more actively by making a mostly dependable system less reliable. It does not describe pointing out errors when decision makers overlook them, and it cautions that more research is needed on catch trials’ efficacy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • If oversight persons receive AI output after making up their own mind, they are not initially affected and may better judge appropriateness.

    “Alternatively, oversight persons may only receive the AI-based output after they have made up their own mind about a specific case because then they would not initially be affected by the AIbased output and may be in a better position to judge whether the AI-based output is appropriate [31, 89].”

    From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [89]

  • Diffusion of responsibility occurs in tasks humans share with automatic systems.

    “Diffusion of responsibility also occurs in tasks humans share with automatic systems”

    From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019

  • Automation bias tends to occur especially when the system is perceived as highly reliable and has been working error-free for a long time (low failure rates).

    “Since automation bias tends to occur especially when the system is perceived as highly reliable (Zerilli et al., 2019) and has been working error-free for a long time, i.e., has low failure rates (Parasuraman and Manzey, 2010), companies should make decision makers aware of possible system failures not once, but on a regular base.”

    From Check the box! How to deal with automation bias in AI-based personnel selection (2023) · cites this paper as Zerilli et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

3 of the 67 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. Amoroso, N., La Rocca, M., Bruno, S., Maggipinto, T., Monaco, A., Bellotti, R., Tangaro, S., the Alzhei- mer's Disease Neuroimaging Initiative. (2017). Brain structural connectivity atrophy in Alzheimer's disease. arXiv :1709.02369 v1. · arXiv
  2. Bagheri, N., & Jamieson, G. A. (2004). Considering subjective trust and monitoring behavior in assess- ing automation-induced "complacency". In D. A. Vicenzi, M. Mouloua, & O. A. Hancock (Eds.), Human performance, situation awareness, and automation: Current research and trends (pp. 54-59). Mahwah, NJ: Erlbaum.
  3. Blomberg, T., Bales, W., Mann, K., Meldrum, R., Nedelec, J. (2010). Validation of the COMPAS risk assessment classification instrument. Center for Criminology and Public Policy Research College 1 3 Algorithmic Decision-Making and the Control Problem of Criminology and Criminal Justice Florida State University. https ://arxiv .org/pdf/1311.2901. pdf.
  4. House of Lords Select Committee on Artificial Intelligence. (2018). AI in the UK: Ready, willing and able? https ://publi catio ns.parli ament .uk/pa/ld201 719/ldsel ect/ldai/100/100.pdf. IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2017). Ethically aligned design (version 2). https ://ethic sinac tion.ieee.org.

Cited by

Sorted by citations

Related

Authors
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C.
Affiliations
  • University of Cambridge (GB)
  • University of Otago (NZ)
Corresponding author
Zerilli, John
ORCID
Venue
Minds and Machines
Publisher
Springer Science and Business Media LLC
Published
2019-12
Volume
29
Issue
4
Pages
555-578
ISSN
  • 0924-6495 (ISSN-L)
  • 1572-8641
DOI
10.1007/s11023-019-09513-7
OpenAlex
W2996655830
Cited by
145
References
67
Citations by year
2026: 25 · 2025: 30 · 2024: 29 · 2023: 18 · 2022: 20 · 2021: 15 · 2020: 7 · 2019: 1
FWCI
13.50
Type
journal article
Language
en
Open access
Licence
Locations
  • Minds and Machines (publishedVersion, cc-by)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • Apollo (cc-by)
Funders
New Zealand Law Foundation (2016/ILP/10)
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
control problems, algorithmic decision-making, human-machine systems, high-stakes decision making, human-machine collaboration, human-machine interface design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review