Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 13 of the 51 sources its claims cite were checked for support.

Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act

Open access
Laux, Johann · 2024 · AI & SOCIETY

Abstract

Human oversight has become a key mechanism for the governance of artificial intelligence (“AI”). Human overseers are supposed to increase the accuracy and safety of AI systems, uphold human values, and build trust in the technology. Empirical research suggests, however, that humans are not reliable in fulfilling their oversight tasks. They may be lacking in competence or be harmfully incentivised. This creates a challenge for human oversight to be effective. In addressing this challenge, this article aims to make three contributions. First, it surveys the emerging laws of oversight, most importantly the European Union’s Artificial Intelligence Act (“AIA”). It will be shown that while the AIA is concerned with the competence of human overseers, it does not provide much guidance on how to achieve effective oversight and leaves oversight obligations for AI developers underdefined. Second, this article presents a novel taxonomy of human oversight roles, differentiated along whether human intervention is constitutive to, or corrective of a decision made or supported by an AI. The taxonomy allows to propose suggestions for improving effectiveness tailored to the type of oversight in question. Third, drawing on scholarship within democratic theory, this article formulates six normative principles which institutionalise distrust in human oversight of AI. The institutionalisation of distrust has historically been practised in democratic governance. Applied for the first time to AI governance, the principles anticipate the fallibility of human overseers and seek to mitigate them at the level of institutional design. They aim to directly increase the trustworthiness of human oversight and to indirectly inspire well-placed trust in AI governance.

Overview

CoChat Summary

TL;DR

The paper argues that effective human oversight of AI under the EU AI Act is underdefined and proposes a taxonomy of first- and second-degree oversight plus six principles of institutionalised distrust to make oversight more trustworthy. It surveys the AI Act and amendments, identifies competence and incentive challenges, and offers democratic-theory design principles rather than empirical validation.

Key findings

  • The EU AI Act (Art. 14) requires high-risk AI to be effectively overseen by natural persons and demands overseers be competent, trained, authorised, and AI-literate, but the proposal does not specify exact qualifications or how effectiveness is achieved; oversight obligations for developers remain underdefined.
  • The paper introduces a taxonomy distinguishing first-degree oversight (human involvement has counterfactual influence and is constitutive of the AI-supported decision) from second-degree oversight (ex post corrective review, audit, or appeal), yielding a two-by-two matrix across fully and partially automated systems.
  • It identifies two challenges—lack of competence and wrong incentives—creating four areas of concern (A–D) for first- and second-degree oversight.
  • Drawing on democratic theory, it condenses Sztompka’s 12 principles into six design principles for institutionalised distrust: justification; periodical mandates; collective decisions; limited competence of institutions; justiciability and accountability; transparency. Table 3 anticipates all applicable principles positively address wrong incentives (C, D), while effects on competence (A, B) are more uncertain; periodical mandates may negatively affect B. Article 14(5) AIA requires at least two natural persons for remote biometric identification.

Methods

  • Conceptual/legal and normative analysis; no empirical sample or data collection.
  • Surveys emerging EU laws of oversight, especially the Commission’s AI Act proposal and Council/Parliament amendments, plus GDPR Article 22 and ISO standards.
  • Develops the oversight taxonomy and six principles by drawing on democratic theory (Sztompka, Braithwaite, Elster) and applies them to AI governance.

Limitations

  • The paper states the principles would not guarantee risk-free AI systems and provide only a general scaffolding; trade-offs among principles are unavoidable and require further normative choices.
  • The paper states generalised assessments of human overseer reliability are unavailable and local use-case testing is necessary; the primary target is public institutions, though the principles may resonate with private-sector AI.
  • Evident from the text: the analysis is conceptual and does not empirically test the proposed principles; it relies on the Commission proposal and amendments because the final AIA text was not decided at the time of writing.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00146-023-01777-z

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

13 Valid · 1 Not valid · 29 Not checked · 70 Own results and interpretations

48 more claims

Create a free account to see all 113 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 18
  • Probably not a substantial claim. 46
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 47 references

Research works printed but not linked: 5

  1. Agrawal A, Gans J, Goldfarb A (2022) Power and prediction: the disruptive economics of artificial intelligence. Harvard Business Review Press, Boston, M.A.
  2. Cranor LF (2008) A framework for reasoning about the human in the loop. Proceedings of the 1st Conference on Usability, Psychology, and Security (UPSEC’08): 1–15.
  3. Hardin R (ed) (2004) Distrust. Russell Sage Foundation series on trust, vol 8. Russell Sage Foundation, New York
  4. Meta (2022a) How review teams work. https://transparency.fb.com/enforcement/detecting-violations/how-review-teams-work/. Accessed 9 Jan 2023.
  5. Sztompka P (2000) Trust, distrust and the paradox of democracy. Polish Polit Sci Yearbook 29(1):5–22

Cited by

See all 127

Metadata

Authors
Laux, J.
Affiliations
University of Oxford (GB)
Corresponding author
Laux, Johann
Venue
AI & SOCIETY
Published
2023-10-06
Volume
39
Issue
6
Pages
2853-2866
ISSN
  • 0951-5666 (ISSN-L)
  • 1435-5655
OpenAlex
W4387409702
Cited by
127
References
47
Citations by year
2026: 75 · 2025: 35 · 2024: 15 · 2023: 1
FWCI
31.44
Type
journal article
Language
en
Open access
Licence
Locations
  • AI & Society (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • British Academy (PF22\220076)
  • Department of Health and Social Care
  • Alfred P. Sloan Foundation (G-2021-16779)
  • Wellcome Trust (223765/Z/21/Z)
  • Luminate Group
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
human oversight of AI, AI governance, EU AI Act, democratic design, normative principles

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act. https://app.cochat.ai/papers/10.1007/s00146-023-01777-z
The paper
Laux, J. (2024). Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act. AI & SOCIETY. https://doi.org/10.1007/s00146-023-01777-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.