The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 13 of the 51 sources its claims cite were checked for support.
Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act
Open accessAbstract
Human oversight has become a key mechanism for the governance of artificial intelligence (“AI”). Human overseers are supposed to increase the accuracy and safety of AI systems, uphold human values, and build trust in the technology. Empirical research suggests, however, that humans are not reliable in fulfilling their oversight tasks. They may be lacking in competence or be harmfully incentivised. This creates a challenge for human oversight to be effective. In addressing this challenge, this article aims to make three contributions. First, it surveys the emerging laws of oversight, most importantly the European Union’s Artificial Intelligence Act (“AIA”). It will be shown that while the AIA is concerned with the competence of human overseers, it does not provide much guidance on how to achieve effective oversight and leaves oversight obligations for AI developers underdefined. Second, this article presents a novel taxonomy of human oversight roles, differentiated along whether human intervention is constitutive to, or corrective of a decision made or supported by an AI. The taxonomy allows to propose suggestions for improving effectiveness tailored to the type of oversight in question. Third, drawing on scholarship within democratic theory, this article formulates six normative principles which institutionalise distrust in human oversight of AI. The institutionalisation of distrust has historically been practised in democratic governance. Applied for the first time to AI governance, the principles anticipate the fallibility of human overseers and seek to mitigate them at the level of institutional design. They aim to directly increase the trustworthiness of human oversight and to indirectly inspire well-placed trust in AI governance.
Overview
CoChat Summary
TL;DR
The paper argues that effective human oversight of AI under the EU AI Act is underdefined and proposes a taxonomy of first- and second-degree oversight plus six principles of institutionalised distrust to make oversight more trustworthy. It surveys the AI Act and amendments, identifies competence and incentive challenges, and offers democratic-theory design principles rather than empirical validation.
Key findings
- The EU AI Act (Art. 14) requires high-risk AI to be effectively overseen by natural persons and demands overseers be competent, trained, authorised, and AI-literate, but the proposal does not specify exact qualifications or how effectiveness is achieved; oversight obligations for developers remain underdefined.
- The paper introduces a taxonomy distinguishing first-degree oversight (human involvement has counterfactual influence and is constitutive of the AI-supported decision) from second-degree oversight (ex post corrective review, audit, or appeal), yielding a two-by-two matrix across fully and partially automated systems.
- It identifies two challenges—lack of competence and wrong incentives—creating four areas of concern (A–D) for first- and second-degree oversight.
- Drawing on democratic theory, it condenses Sztompka’s 12 principles into six design principles for institutionalised distrust: justification; periodical mandates; collective decisions; limited competence of institutions; justiciability and accountability; transparency. Table 3 anticipates all applicable principles positively address wrong incentives (C, D), while effects on competence (A, B) are more uncertain; periodical mandates may negatively affect B. Article 14(5) AIA requires at least two natural persons for remote biometric identification.
Methods
- Conceptual/legal and normative analysis; no empirical sample or data collection.
- Surveys emerging EU laws of oversight, especially the Commission’s AI Act proposal and Council/Parliament amendments, plus GDPR Article 22 and ISO standards.
- Develops the oversight taxonomy and six principles by drawing on democratic theory (Sztompka, Braithwaite, Elster) and applies them to AI governance.
Limitations
- The paper states the principles would not guarantee risk-free AI systems and provide only a general scaffolding; trade-offs among principles are unavoidable and require further normative choices.
- The paper states generalised assessments of human overseer reliability are unavailable and local use-case testing is necessary; the primary target is public institutions, though the principles may resonate with private-sector AI.
- Evident from the text: the analysis is conceptual and does not empirically test the proposed principles; it relies on the Commission proposal and amendments because the final AIA text was not decided at the time of writing.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 Valid · 1 Not valid · 29 Not checked · 70 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Due to automation bias, people may overestimate an AI’s accuracy and consistency in its performance (Jones-Jang and Park 2022: 2).”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Two challenges for human oversight
What the sources say
- Supportedcites Jones-Jang and Park 2022 source
The section “Heuristic #1: automation bias and algorithmic aversion” states that automation bias occurs when users overestimate AI’s consistent performance and accuracy. This directly supports the citing sentence’s paraphrase. (LLM, conf=0.95)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Human oversight as a governance mechanisms can thus give a false sense of security if humans fail systematically at overseeing AI (cf. (Green 2022: 7)).”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Two challenges for human oversight
What the sources say
- Supportedcites Green 2022 source
Section 4.2.1 explicitly says that assuming effective human oversight provides a false sense of security when adopting algorithms. The introductory passage likewise says that if people cannot oversee algorithms as intended, oversight policies may alleviate scrutiny without addressing the underlying concerns. The paper discusses this mechanism in the context of government algorithms. (LLM, conf=0.92)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“By adding members with different life experiences and cognitive problem-solving skills, and given favourable conditions, diverse groups may even outperform experts (Hong and Page 2004).”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Institutionalising distrust in AI oversight > Six design principles for human oversight
What the sources say
- Supportedcites Hong and Page 2004 source
The Abstract says that experiences can shape problem-solving perspectives and tools, while the paper specifically models differences in how people solve problems. The Concluding Remarks state that, under identified conditions, a diverse group of problem solvers can outperform a group of the best problem solvers. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When a public institution implements a novel AI system, this can change its perceived trustworthiness and the processes by which it is able to generate citizens’ trust in its actions ([Laux et. al 2023b: 8]; drawing on: Bodó, 2021: 2675).”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Institutionalising distrust in AI oversight > The democratic theory of institutionalised distrust
What the sources say
- Not matchedcites Laux et. al 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Bodó, 2021 source
The ‘Trust by technology’ section says that when public entities use machine-learning tools, their operations change, affecting their trustworthiness and ability to produce trust. The supplied passage does not identify a page number. (LLM, conf=0.93)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“The aversion has been evidenced in both lay people and experts (Dietvorst et al. 2015: 114).”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Two challenges for human oversight
What the sources say
- Supportedcites Dietvorst et al. 2015 source
The introduction (p. 1) explicitly says that experts and laypeople remain resistant to using algorithms, often preferring inferior human forecasts. It identifies this pattern as algorithm aversion, supporting the claim for both groups. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
48 more claims
Create a free account to see all 113 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 46
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research has shown that such tools may produce insecure or suboptimal code, requiring users to review and revise outputs carefully to avoid compounding mistakes (Groothuijsen et al., 2024;Laux, 2024;Nguyen & Nadi, 2022).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Laux, 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings reinforce the view that, despite the sophistication of GenAI tools, human oversight and technical expertise remain essential for ensuring code accuracy and functionality (Garg et al., 2025;Laux, 2024;Zhang et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Laux, 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Importantly, the role of human oversight as a safeguard must itself be treated with caution: rather than a straightforward solution, it represents a complex governance challenge shaped by institutional design, competing logics of trust and distrust, and the distributed nature of control in AI systems (Laux 2024; Maggetti 2026; Foth 2026).”
From Digital civics in action: the promise and emerging practices of generative AI in grassroots community advocacy (2026) · cites this paper as Laux 2024
The cited source says what the claim attributes to it.
In the citing paper
“However, this standardization process is still in its early stages and is limited to a few general guidelines [49].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Introduction
“This article formulates six principles for the institutionalisation of distrust in human oversight: justification; periodical mandates; collective decisions; limited competence of institutions; justiciability and accountability; transparency.”
§ Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act > Institutionalising distrust in AI oversight > Results
“| 1. Legitimacy | + | + | + | + |”
Citations
Cites
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 2×
“The aversion has been evidenced in both lay people and experts (Dietvorst et al. 2015: 114).”
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“Automated decision-making systems have long been associated with opacity and “black-box” decisions (Burrell 2016; Pasquale 2015).”
Bainbridge, L. (1983) · Automatica · doi:10.1016/0005-1098(83)90046-8 · Cited by 2,205 · cited 1×
“Add empirical research such as the aforementioned work on automation bias to the picture, and some argue that overseeing automated systems is an impossible assignment for humans ((Green 2022: 11); citing: (Bainbridge 1983)).”
Groups of diverse problem solvers can outperform groups of high-ability problem solvers
Hong, L., & Page, S. E. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403723101 · Cited by 2,103 · cited 1×
“By adding members with different life experiences and cognitive problem-solving skills, and given favourable conditions, diverse groups may even outperform experts (Hong and Page 2004).”
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · cited 1×
“Again, some commentators doubt that additional training can alleviate the gap between humans and AI in the expected average decisional quality, at least along some parameters ((Green 2022: 11–12); citing: (Parasuraman and Manzey 2010; Skitka et al. 1999)).”
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Page, S. (2008) · Princeton University Press eBooks · doi:10.1515/9781400830282 · Cited by 881
Research works printed but not linked: 5
- Agrawal A, Gans J, Goldfarb A (2022) Power and prediction: the disruptive economics of artificial intelligence. Harvard Business Review Press, Boston, M.A.
- Cranor LF (2008) A framework for reasoning about the human in the loop. Proceedings of the 1st Conference on Usability, Psychology, and Security (UPSEC’08): 1–15.
- Hardin R (ed) (2004) Distrust. Russell Sage Foundation series on trust, vol 8. Russell Sage Foundation, New York
- Meta (2022a) How review teams work. https://transparency.fb.com/enforcement/detecting-violations/how-review-teams-work/. Accessed 9 Jan 2023.
- Sztompka P (2000) Trust, distrust and the paradox of democracy. Polish Polit Sci Yearbook 29(1):5–22
Cited by
Is human oversight to AI systems still possible?
Holzinger, A., Zatloukal, K., & Müller, H. (2024) · New Biotechnology · doi:10.1016/j.nbt.2024.12.003 · Cited by 179
On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659051 · Cited by 91
AI Governance: A Systematic Literature ReviewPreprint
Batool, A., Zowghi, D., & Bano, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4784792/v1 · Cited by 70
Laux, J., & Ruschemeier, H. (2025) · European Journal of Risk Regulation · doi:10.1017/err.2025.10033 · Cited by 69
Self-citationWu, H., Li, D., & Mo, X. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13312-8 · Cited by 62
“For instance, research indicates that cultural values and societal norms significantly shape individuals’ perceptions and acceptance of AI technologies (Koenig, 2024; Laux, 2023).”
Metadata
- Authors
- Laux, J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Laux, Johann
- ORCID
- Laux, Johann
- Venue
- AI & SOCIETY
- Published
- 2023-10-06
- Volume
- 39
- Issue
- 6
- Pages
- 2853-2866
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- PMC
- PMC11614927
- OpenAlex
- W4387409702
- Cited by
- 127
- References
- 47
- Citations by year
- 2026: 75 · 2025: 35 · 2024: 15 · 2023: 1
- FWCI
- 31.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- British Academy (PF22\220076)
- Department of Health and Social Care
- Alfred P. Sloan Foundation (G-2021-16779)
- Wellcome Trust (223765/Z/21/Z)
- Luminate Group
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- human oversight of AI, AI governance, EU AI Act, democratic design, normative principles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act. https://app.cochat.ai/papers/10.1007/s00146-023-01777-z
- The paper
- Laux, J. (2024). Institutionalised distrust and human oversight of artificial intelligence: towards a democratic design of AI governance under the European Union AI Act. AI & SOCIETY. https://doi.org/10.1007/s00146-023-01777-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.