Reciprocal trust and distrust in artificial intelligence systems: the hard problem of regulation
Open accessThe DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Abstract
Policymakers, scientists, and the public are increasingly confronted with thorny questions about the regulation of artificial intelligence (AI) systems. A key common thread concerns whether AI can be trusted and the factors that can make it more trustworthy in front of stakeholders and users. This is indeed crucial, as the trustworthiness of AI systems is fundamental for both democratic governance and for the development and deployment of AI. This article advances the discussion by arguing that AI systems should also be recognized, at least to some extent, as artifacts capable of exercising a form of agency, thereby enabling them to engage in relationships of trust or distrust with humans. It further examines the implications of these reciprocal trust dynamics for regulators tasked with overseeing AI systems. The article concludes by identifying key tensions and unresolved dilemmas that these dynamics pose for the future of AI regulation and governance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Importantly, the role of human oversight as a safeguard must itself be treated with caution: rather than a straightforward solution, it represents a complex governance challenge shaped by institutional design, competing logics of trust and distrust, and the distributed nature of control in AI systems (Laux 2024; Maggetti 2026; Foth 2026).”
From Digital civics in action: the promise and emerging practices of generative AI in grassroots community advocacy (2026) · cites this paper as Maggetti 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Can Quantum-Mechanical Description of Physical Reality Be Considered Complete?
Einstein, A., Podolsky, B., & Rosen, N. (1935) · Physical Review · doi:10.1103/physrev.47.777 · Cited by 16,654 · cited 1×
“Thought experiments are famously employed in physics (Einstein et al. 1935; Schrödinger 1935) and in moral philosophy (Singer 1972; Thomson 1984), but they may serve as valuable analytical tools in political science and regulatory governance as well.”
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“Classic models in psychology and organizational studies highlight key attributes that make a trustee trustworthy, typically some combination of predictable competence, benevolence, and integrity, as articulated in the ABI framework (Mayer et al. 1995).”
Not So Different After All: A Cross-Discipline View Of Trust
Rousseau, D. M., Sitkin, S. B., Burt, R. S., & Camerer, C. (1998) · Academy of Management Review · doi:10.5465/amr.1998.926617 · Cited by 10,274 · cited 2×
“…can nevertheless be understood as functionally equivalent to trust behaviors— even if not phenomenologically identical—particularly the subtype Rousseau et al. (1998, p. 399) term “calculus-based trust”: a rational, incentive-driven evaluation without psychological and affective components.”
Trust And Distrust: New Relationships and Realities
Lewicki, R. J., McAllister, D. J., & Bies, R. J. (1998) · The Academy of Management Review · doi:10.5465/amr.1998.926620 · Cited by 3,091 · cited 1×
“Accord-ingly, it has been argued that one can simultaneously trust and distrust the same entity in different respects (Lewicki et al. 1998).”
Capoccia, G., & Kelemen, R. D. (2007) · World Politics · doi:10.1017/s0043887100020852 · Cited by 2,410 · cited 1×
“…has also been adopted by comparative historical institutional scholars to deal with the concepts of path dependence and critical junctures, emphasizing that claims regarding institutional lock-in require the explicit identification of credible alternative trajectories (Capoccia and Kelemen 2007).”
Thomson, J. J. (1985) · The Yale Law Journal · doi:10.2307/796133 · Cited by 1,304 · cited 1×
“Thought experiments are famously employed in physics (Einstein et al. 1935; Schrödinger 1935) and in moral philosophy (Singer 1972; Thomson 1984), but they may serve as valuable analytical tools in political science and regulatory governance as well.”
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038
Human-in-the-loop machine learning: a state of the art
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10246-w · Cited by 1,025
Managing artificial intelligence
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16274 · Cited by 898 · cited 1×
“Second, the autonomy of the system could be tied to codified invariants (Berente et al. 2021; Wang 2021).”
Algorithmic regulation: A critical interrogation
Yeung, K. (2018) · Regulation & Governance · doi:10.1111/rego.12158 · Cited by 842 · cited 1×
“Yet , precisely because they rely heavily on feed-back, AI systems may also exhibit behaviors analogous to distrust when confronted with conflicting, incomplete, systematically biased, or unreliable inputs (Kordzadeh and Ghasemaghaei 2022; Yeung 2018).”
Die gegenw�rtige Situation in der Quantenmechanik
Schr�dinger, E. (1935) · Die Naturwissenschaften · doi:10.1007/bf01491987 · Cited by 832 · cited 1×
“Thought experiments are famously employed in physics (Einstein et al. 1935; Schrödinger 1935) and in moral philosophy (Singer 1972; Thomson 1984), but they may serve as valuable analytical tools in political science and regulatory governance as well.”
A systematic review of algorithm aversion in augmented decision making
Burton, J. W., Stein, M., & Jensen, T. B. (2020) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2155 · Cited by 814
Algorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770 · cited 1×
“Yet , precisely because they rely heavily on feed-back, AI systems may also exhibit behaviors analogous to distrust when confronted with conflicting, incomplete, systematically biased, or unreliable inputs (Kordzadeh and Ghasemaghaei 2022; Yeung 2018).”
The Government of Risk: Understanding Risk Regulation Regimes
Hood, C., Rothstein, H., & Baldwin, R. (2001) · doi:10.1093/0199243638.001.0001 · Cited by 760
Counterfactuals and Hypothesis Testing in Political Science
Fearon, J. D. (1991) · World Politics · doi:10.2307/2010470 · Cited by 729 · cited 1×
“…and comparative political scientists were already implicitly testing hypotheses against imagined alternative outcomes, James Fearon observed that the primary problem was the absence of clear standards to determine which hypothetical scenarios could provide cred-ible evidence (Fearon 1991).”
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709 · cited 2×
“In this emerging scholarship, attributes such as reliability, robustness, safety, fairness, explainability, and security are frequently cited as prereq-uisites for an AI system to be deemed trustworthy (Chatila et al. 2021; Kaur et al. 2022; Li et al. 2023; Liang et al. 2022).”
Accountable Artificial Intelligence: Holding Algorithms to Account
Busuioc, M. (2021) · Public Administration Review · doi:10.1111/puar.13293 · Cited by 652 · cited 1×
“As a matter of fact, the design, adoption, deployment, and use of AI systems is a deeply political process, which does not only reallocate decision-making authority, but also reshapes accountability relations among developers, users, targets, and regulators (Busuioc 2021).”
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · cited 2×
“In this emerging scholarship, attributes such as reliability, robustness, safety, fairness, explainability, and security are frequently cited as prereq-uisites for an AI system to be deemed trustworthy (Chatila et al. 2021; Kaur et al. 2022; Li et al. 2023; Liang et al. 2022).”
What influences algorithmic decision-making? A systematic literature review on algorithm aversion
Mahmud, H., Islam, A. K. M. N., Ahmed, S. I., & Smolander, K. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121390 · Cited by 601 · cited 1×
“On the other hand, studies on “algorithm aversion” find that people often abandon or underutilize algorithmic advice after seeing even minor errors, preferring fallible human judgment instead (Burton et al. 2020; Mahmud et al. 2022; Sunstein and Gaffe 2024).”
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559 · cited 2×
“In this emerging scholarship, attributes such as reliability, robustness, safety, fairness, explainability, and security are frequently cited as prereq-uisites for an AI system to be deemed trustworthy (Chatila et al. 2021; Kaur et al. 2022; Li et al. 2023; Liang et al. 2022).”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545 · cited 1×
“But scholars are increasingly recognizing the bidirectional nature of trust in socio-technical systems (Bareis 2024; Jacovi et al. 2021; Lee-Geiller 2024; Sagona et al. 2025; Wang et al. 2024).”
Trust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507 · cited 1×
“Empirical research in human–AI interaction provides evidence that the level and appropriateness of trust directly influence decision quality in public-sector and high-stakes contexts (Afroogh et al. 2024).”
Ethics in artificial intelligence: introduction to the special issue
Dignum, V. (2018) · Ethics and Information Technology · doi:10.1007/s10676-018-9450-z · Cited by 459 · cited 1×
“First, non-overridable safety interlocks may be implemented (Dignum 2018; Nordland 2004).”
Viewpoint: Human-in-the-loop Artificial Intelligence
Zanzotto, F. M. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11345 · Cited by 380
Validating a Scale for Citizen Trust in Government Organizations
Grimmelikhuijsen, S., & Knies, E. (2017) · International Review of Administrative Sciences · doi:10.1177/0020852315585950 · Cited by 366 · cited 1×
“…governance work extends these insights by showing that public institutions earn trust through procedural (transparency, accountability), quality (competence, reliability), and outcome dimensions (effectiveness, justice, public welfare) (Grimmelikhuijsen and Knies 2017; Six and Verhoest 2017).”
Managing extreme AI risks amid rapid progress
Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., et al. (2024) · Science · doi:10.1126/science.adn0117 · Cited by 349 · cited 1×
“…objectives explicitly specified by designers, meaning that AI systems can evolve and develop unforeseen patterns of trust and distrust through their own autonomous learning processes—patterns that their programmers neither explicitly specified nor necessarily anticipated (Bengio et al. 2024, 2023).”
Warren, M. E. (1999) · Democracy and Trust · doi:10.1017/cbo9780511659959.011 · Cited by 305
How transparency modulates trust in artificial intelligence
Zerilli, J., Bhatt, U., & Weller, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100455 · Cited by 257 · cited 1×
“Researchers have indeed documented cases in which human decision-makers deferred to an AI despite obvious errors or biases, highlighting that an “impressed” user may ignore contradictory evidence due to overestimation of the AI’s competence or impartiality (Zerilli et al. 2022).”
On human-in-the-loop optimization of human–robot interaction
Slade, P., Atkeson, C. G., Donelan, J. M., Houdijk, H., Ingraham, K. A., Kim, M., Kong, K., Poggensee, K. L., Riener, R., Steinert, M., et al. (2024) · Nature · doi:10.1038/s41586-024-07697-2 · Cited by 199 · cited 1×
“Third, and above all, there should be human veto power with protected decision space (Mosqueira-Rey et al. 2023; Slade et al. 2024; Zanzo-tto 2019).”
Artificial Intelligence Programming
Charniak, E., Riesbeck, C. K., McDermott, D. V., & Meehan, J. R. (2014) · Psychology Press eBooks · doi:10.4324/9781315802251 · Cited by 179
Lukyanenko, R., Maass, W., & Storey, V. C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00605-4 · Cited by 171 · cited 1×
“This concept, popularized by the European Union (EU) High-Level Expert Group’s 2019 guidelines, encapsulates the idea that AI should be designed to meet certain ethical and technical standards so that it deserves trust (Lukyanenko et al. 2022).”
Laux, J. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01777-z · Cited by 127
The Long-Run Consequences of Chernobyl: Evidence on Subjective Well-Being, Mental Health and Welfare
Danzer, A. M., & Danzer, N. (2014) · Journal of Public Economics · doi:10.1016/j.jpubeco.2016.01.001 · Cited by 126 · cited 1×
“…limited mandate, the AI’s timely and precise warnings were functionally irrelevant to preventing the disaster, without any significant variations in terms of catastrophic outcomes with respect to the historical case (Bard et al. 1997; Danzer and Danzer 2016; Rytömaa 1996; Yablokov et al. 2010).”
Chatila, R., Dignum, V., Fisher, M., Giannotti, F., Morik, K. J., Russell, S., & Yeung, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69128-8_2 · Cited by 113
AI‐assisted scenario generation for strategic planning
Spaniol, M. J., & Rowland, N. J. (2023) · Futures & Foresight Science · doi:10.1002/ffo2.148 · Cited by 103 · cited 1×
“To systematically check for the plausibility of the scenarios (Bresinsky and Hager 2024; Spaniol and Rowland 2023), a very concise, ex-post version of the counterfactuals has been prompted into the main frontier LLMs at the time of writing, without providing any additional background (see the…”
What Is Consciousness, and Could Machines Have It?
Dehaene, S., Lau, H. C., & Kouider, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-54173-6_4 · Cited by 102
Artificial intelligence in safety-critical systems: a systematic review
Wang, Y., & Chung, S. H. (2022) · Industrial Management & Data Systems · doi:10.1108/imds-07-2021-0419 · Cited by 96 · cited 1×
“Third, dual-key human override could be helpful (Wang and Chung 2022).”
Avramova, M. N., & Ivanov, K. (2010) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2010.03.009 · Cited by 90 · cited 1×
“Second, AI systems should be transparent about uncertainty (Avramova and Ivanov 2010).”
Chernobyl, 10 Years After: Health Consequences
Bard, D., Verger, P., & Hubert, P. (1997) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a017952 · Cited by 89 · cited 1×
“…limited mandate, the AI’s timely and precise warnings were functionally irrelevant to preventing the disaster, without any significant variations in terms of catastrophic outcomes with respect to the historical case (Bard et al. 1997; Danzer and Danzer 2016; Rytömaa 1996; Yablokov et al. 2010).”
Trust in AI-assisted health systems and AI’s trust in humans
Sagona, M., Dai, T., Macis, M., & Darden, M. (2025) · npj Health Systems · doi:10.1038/s44401-025-00016-5 · Cited by 71 · cited 3×
“Sagona et al. (2025) also note that an AI may “struggle to contextualize” human actions without some model of trust—for instance, if a human operator overrides an AI recommendation, an unso-phisticated AI might interpret that as an error in the human’s behavior rather than a correction to the AI,…”
Impact of Model Interpretability and Outcome Feedback on Trust in AI
Ahn, D., Almaatouq, A., Gulabani, M., & Hosanagar, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642780 · Cited by 69 · cited 1×
“Both excessive trust and excessive distrust can undermine performance and legitimacy, while appropriately aligned trust (trusting a trustee to the extent it is trustworthy) tends to enhance effectiveness (Ahn et al. 2024).”
Trust, trustworthiness and AI governance
Lahusen, C., Maggetti, M., & Slavkovik, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-71761-0 · Cited by 68 · cited 4×
“AI governance involves indeed distinct, yet interlocked problems of trustworthiness (Lahusen et al. 2024): trustworthiness must be considered at the level of the algorithm (AI outputs and processes), of the human operators or decision-makers using the AI, and of the institutions regulating AI…”
InfluentialSelf-citationThe necessity of AI audit standards boards
Manheim, D., Martin, S., Bailey, M., Samin, M., & Greutzmacher, R. (2025) · AI & Society · doi:10.1007/s00146-025-02320-y · Cited by 68 · cited 1×
“More innovatively, there is an ongoing discussion about creating new AI oversight regulatory agencies with authority to license high-risk AI systems and conduct audits (Guha et al. 2024; Kaminski 2023; Manheim et al. 2025; Stewart 2024).”
Formation of Hot Particles During the Chernobyl Nuclear Power Plant Accident
Kashparov, V., Ivanov, Y. A., Zvarisch, S. I., Protsak, V., Khomutinin, Y., Kurepin, A., & Pazukhin, É. M. (1996) · Nuclear Technology · doi:10.13182/nt96-a35253 · Cited by 63 · cited 1×
“…due to the onset of xenon poisoning, occurring as the xenon-135 concentration (a byproduct of nuclear fission, which absorbs neutrons and inhibits the nuclear chain reaction) started to increase faster than it could be burned off (Grishanin 2010; Kashparov et al. 1996; Mer-cier et al. 2021).”
Pitfalls in implementing a strategic early warning system
Schwarz, J. O. (2005) · foresight · doi:10.1108/14636680510611813 · Cited by 62
Lebow, R. N., & Stein, J. G. (1995) · Political Science Quarterly · doi:10.2307/2152358 · Cited by 61 · cited 1×
“…unfolding between the U.S. and USSR, which might have apparently escalated in that period, but which would have, in practice, hardly resulted in a first strike by the U.S., against a background where nuclear weapons primarily hold a deterrence strategic function (Lebow and Stein 1995).”
Duenser, A., & Douglas, D. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3322586 · Cited by 59 · cited 1×
“Accordingly, recent calls in the explainable AI (XAI) community urge increasing trust in trustworthy AI and increasing distrust in untrustworthy AI (Duenser and Douglas 2023; Scharowski et al. 2024).”
Trust in context: The impact of regulation on blockchain and DeFi
Bodó, B., & Filippi, P. D. (2024) · Regulation & Governance · doi:10.1111/rego.12637 · Cited by 57
Verhoest, K., Maggetti, M., Guaschino, E., & Wynen, J. (2024) · Regulation & Governance · doi:10.1111/rego.12596 · Cited by 51 · cited 4×
“Distrust, while initially often treated as the inverse of trust, has been reconceptualized in recent scholarship as a distinct and sometimes positive force (Mag-getti et al. 2023; Verhoest et al. 2025).”
InfluentialSelf-citationDesign and Piloted Simulator Evaluation of Adaptive Safe Flight Envelope Protection Algorithm
Lombaerts, T., Looye, G., Ellerbroek, J., & y Martin, M. R. (2017) · Journal of Guidance Control and Dynamics · doi:10.2514/1.g002525 · Cited by 47 · cited 1×
“…especially in areas where a false negative is catastrophic—nuclear-plant control, chemical reactors, flight-envelope protection (Dong et al. 2023; Lombaerts et al. 2017)—regulation might be considered with the aim to empower the AI system to over-rule untrustworthy, potentially unsafe human…”
Resilient Flight Control: An Architecture for Human Supervision of Automation
Farjadian, A. B., Thomsen, B. T., Annaswamy, A. M., & Woods, D. D. (2020) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2019.2959542 · Cited by 41 · cited 1×
“For example, aviation regulators have long required that autopilot systems defer to human pilots: humans must supervise autopilot and can intervene at any time (Farjadian et al. 2020).”
The trustification of AI. Disclosing the bridging pillars that tie trust and AI together
Bareis, J. (2024) · Big Data & Society · doi:10.1177/20539517241249430 · Cited by 40 · cited 1×
“But scholars are increasingly recognizing the bidirectional nature of trust in socio-technical systems (Bareis 2024; Jacovi et al. 2021; Lee-Geiller 2024; Sagona et al. 2025; Wang et al. 2024).”
Review on the Recent Progress in Nuclear Plant Dynamical Modeling and Control
Dong, Z., Cheng, Z., Zhu, Y., Huang, X., Dong, Y., & Zhang, Z. (2023) · Energies · doi:10.3390/en16031443 · Cited by 37 · cited 1×
“…observation that, especially in areas where a false negative is catastrophic—nuclear-plant control, chemical reactors, flight-envelope protection (Dong et al. 2023; Lombaerts et al. 2017)—regulation might be considered with the aim to empower the AI system to over-rule untrustworthy,…”
Bao, H., Liu, W., & Dai, Z. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124102 · Cited by 37 · cited 1×
“…public administrators across different tasks, highlighting that trust levels are largely context-dependent: respondents showed higher trust in an AI for highly computational tasks (e.g., data analysis or auditing) but preferred a human for more conversational or advisory tasks (Bao et al. 2025).”
Denton, H. R. (1987) · Annals of Nuclear Energy · doi:10.1016/0306-4549(87)90132-0 · Cited by 33
Artificial Intelligence and Nuclear Command and Control
Fitzpatrick, M. (2019) · Survival · doi:10.1080/00396338.2019.1614782 · Cited by 26 · cited 1×
“…from the first coun-terfactual scenario is that especially in systems where a false positive is overwhelmingly costly—nuclear command, strategic early-warning, cybersecurity attacks (Fitzpatrick 2019; Schwarz 2005)—regulation is needed to slow machines down and keep humans with time to think.”
Hardin, R. (2002) · European Review · doi:10.1017/s1062798702000078 · Cited by 26 · cited 1×
“Indeed, a certain degree of distrust—in the sense of skepticism, vigilance, or watchfulness—can be socially beneficial: distrust—when channeled through proper oversight—serves to prevent misbehavior and keep institutions in check (Hardin 2002; Sztompka 1999; Warren 1999).”
Rytömaa, T. (1996) · Annals of Medicine · doi:10.3109/07853899609092930 · Cited by 21 · cited 1×
“…limited mandate, the AI’s timely and precise warnings were functionally irrelevant to preventing the disaster, without any significant variations in terms of catastrophic outcomes with respect to the historical case (Bard et al. 1997; Danzer and Danzer 2016; Rytömaa 1996; Yablokov et al. 2010).”
Forden, G. E., Подвиг, П., & Postol, T. A. (2000) · IEEE Spectrum · doi:10.1109/6.825657 · Cited by 19 · cited 5×
“The absence of intuition-based reasoning prevents the AI system from recognizing the incongruence of a U.S. strike at that time, specifically involving five missiles only—as the system was designed to consider even a single intercontinental ballistic missile as an existential threat.”
InfluentialSimulation of the chernobyl accident
Fletcher, C. D., Chambers, R., Bolander, M. A., & Dallman, R. J. (1988) · Nuclear Engineering and Design · doi:10.1016/0029-5493(88)90337-8 · Cited by 17 · cited 1×
“In addition, two automatic shutdown circuits (AZ-1 and AZ-2), designed to trigger an emergency shutdown (SCRAM) under unsafe conditions, were disabled to avoid interference with the planned test sequence (Chao et al. 1988; Fletcher et al. 1988).”
When safe enough is not good enough: Organizing safety at Chernobyl
Schmid, S. D. (2011) · Bulletin of the Atomic Scientists · doi:10.1177/0096340211399404 · Cited by 16 · cited 1×
“During this rapid escalation, the AI system registered neutron flux levels exceeding design limits by a factor of twenty and automatically produced a final warning message (Malko 2002; Schmid 2011; Weber et al. 1987).”
Shekhar, S., & Vold, P. (2020) · The MIT Press eBooks · doi:10.7551/mitpress/11275.001.0001 · Cited by 16 · cited 1×
“…on 26 September 1983, the Soviet satellite-based early-warning system (Oko) at Serpukhov-15 control center, located approximately 120 km south of Moscow, incorrectly detected five intercon-tinental ballistic missile launches from the continental U.S (Forden et al. 2000; Shekhar and Vold 2020).”
Pitale, M., Abbaspour, A., & Upadhyay, D. K. (2024) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2024-01-2864 · Cited by 16
Emerging structures of control for algorithms on the Internet
Saurwein, F. (2019) · Routledge eBooks · doi:10.4324/9781351115780-13 · Cited by 12 · cited 1×
“Scholars argue that shared or “distributed” accountability frameworks are needed in such cases, reflecting the joint role of human and AI systems (Basti and Vitiello 2023; Saurwein 2019).”
A simplified analysis of the Chernobyl accident
Mercier, B., Yang, D., Zhuang, Z., & Liang, J. (2021) · EPJ Nuclear Sciences & Technologies · doi:10.1051/epjn/2020021 · Cited by 11
Deep Learning Opacity, and the Ethical Accountability of AI Systems. A New Perspective
Basti, G., & Vitiello, G. (2023) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-031-39113-2_2 · Cited by 10
The role of chemical reactions in the Chernobyl accident
Grishanin, E. I. (2010) · Physics of Atomic Nuclei · doi:10.1134/s1063778810140073 · Cited by 9 · cited 1×
“…due to the onset of xenon poisoning, occurring as the xenon-135 concentration (a byproduct of nuclear fission, which absorbs neutrons and inhibits the nuclear chain reaction) started to increase faster than it could be burned off (Grishanin 2010; Kashparov et al. 1996; Mer-cier et al. 2021).”
Zwald, Z., Kennedy, R., & Ozer, A. L. (2025) · Journal of Peace Research · doi:10.1177/00223433241290885 · Cited by 9 · cited 1×
“…may judge human decision-makers differently when AI is involved and tend to place extra blame on judges who use AI advice in sentencing if a mis-take occurs, compared to judges who did not use AI—even if the judge ultimately made the decision themselves (Ozer et al. 2024; Zwald et al. 2024).”
The Paradox of Algorithms and Blame on Public Decision-makers
Ozer, A. L., Waggoner, P., & Kennedy, R. (2024) · Business and Politics · doi:10.1017/bap.2023.35 · Cited by 8 · cited 1×
“…may judge human decision-makers differently when AI is involved and tend to place extra blame on judges who use AI advice in sentencing if a mis-take occurs, compared to judges who did not use AI—even if the judge ultimately made the decision themselves (Ozer et al. 2024; Zwald et al. 2024).”
Pentagon Moves Toward First-Strike Capability
Smith, R. J. (1982) · Science · doi:10.1126/science.216.4546.596 · Cited by 7 · cited 1×
“Thereby, operating without human intervention, the AI autonomously triggers an immediate retaliatory protocol consistent with Soviet strategic doctrine of launch-on-warning (Jacobsen 1990; Krepon and Perkovich; Smith 1982).”
An Analysis of the Chernobyl Accident Using RETRAN-02/MOD3
Chao, J., Chexal, V. K., Layman, W. H., Vine, G., Jensen, P. J., & Dastur, A. R. (1988) · Nuclear Technology · doi:10.13182/nt88-a34142 · Cited by 6 · cited 1×
“In addition, two automatic shutdown circuits (AZ-1 and AZ-2), designed to trigger an emergency shutdown (SCRAM) under unsafe conditions, were disabled to avoid interference with the planned test sequence (Chao et al. 1988; Fletcher et al. 1988).”
Empowered but Challenged: The Impact of Task Expansion on Data Protection Authorities
Maggetti, M., Salihi, B., Pagliarin, E., & Vagionaki, T. (2025) · Policy & Internet · doi:10.1002/poi3.70001 · Cited by 6 · cited 1×
“Data protection authorities have also taken over regulatory tasks in areas related to the use of AI systems, especially in the public sector (Maggetti et al. 2025).”
Self-citationAnalysis of Reactivity Coefficients of Chernobyl Reactor by Cell Calculation
Tsuchihashi, K., & AKINO, F. (1987) · Journal of Nuclear Science and Technology · doi:10.1080/18811248.1987.9733540 · Cited by 5
Reactivity Coefficients in Nuclear Reactors
Hyland, M. (1987) · Europhysics news · doi:10.1051/epn/19871811133 · Cited by 4 · cited 1×
“Due to the extensively with-drawn control rods, the sudden increase in steam formation rapidly raised reactivity, triggering a runaway reaction as predicted by the reactor’s positive-void coefficient (Hyland 1987).”
One-Dimensional Considerations on the Initial Phase of the Chernobyl Accident
Vanttola, T. A., & Rajamäki, M. K. (1989) · Nuclear Technology · doi:10.13182/nt89-a34225 · Cited by 3 · cited 1×
“This AI-based monitoring program continuously tracked key safety parameters, including coolant flow rates, control rod positions, and the reactor’s positive-void coefficient (Tsuchi-hashi and Akino 1987; Vanttola and Rajamäki 1989).”
Can Artificial Intelligence Be Safe?
Nordland, O. (2004) · Springer eBooks · doi:10.1007/978-0-85729-410-4_65 · Cited by 3 · cited 1×
“First, non-overridable safety interlocks may be implemented (Dignum 2018; Nordland 2004).”
Soviet Strategic Policy Since 1945
Jacobsen, C. G. (1990) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-20574-5_7 · Cited by 3 · cited 1×
“Thereby, operating without human intervention, the AI autonomously triggers an immediate retaliatory protocol consistent with Soviet strategic doctrine of launch-on-warning (Jacobsen 1990; Krepon and Perkovich; Smith 1982).”
Chernobyl lessons learned review of N Reactor
Westinghouse Hanford Co., R., Weber, E., USDOE, McNeece, J. P., Omberg, R. P., Stepnewski, D. D., Lutz, R., Henry, R., Bonser, K., & Miller, N. (1987) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/719193 · Cited by 2 · cited 1×
“During this rapid escalation, the AI system registered neutron flux levels exceeding design limits by a factor of twenty and automatically produced a final warning message (Malko 2002; Schmid 2011; Weber et al. 1987).”
Enhanced Flight Envelope Protection: A Novel Reinforcement Learning Approach
Çatak, A., Altunkaya, E. Ç., Demir, M., Koyuncu, E., & Özkol, İ. (2024) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2025.01.182 · Cited by 2
Lee-Geiller, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5063392 · Cited by 2
Strangelove and the ring of truth
Bizony, P. (2014) · Engineering & Technology · doi:10.1049/et.2014.0308 · Cited by 1 · cited 1×
“Subsequent investigations confirmed indeed that sunlight reflecting off high-altitude clouds had produced infrared signals incorrectly identified by sensors as missile launches (Bizony 2014).”
Maggetti, M. (2025) · Edward Elgar Publishing eBooks · doi:10.4337/9781035302840.00011 · Cited by 1
Self-citationPhantom braking in automated vehicles: A theoretical outline and cycling simulator demonstration
Berge, S. H., de Winter, J., Feng, Y., & Hagenzieker, M. (2024) · doi:10.13140/rg.2.2.28681.45921 · Cited by 1
Research works printed but not linked: 32
- Anisimov, Aleksey Pavlovich, and Anatoliy Jakovlevich Ryzhenkov. 2016. "Thirty years after the accident at the Chernobyl nuclear power plant: historical causes, lessons and legal effects." Journal of Energy & Natural Resources Law 34(3):265-83.
- Bengio, Yoshua, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, and Shai Shalev-Shwartz. 2024. "Managing extreme AI risks amid rapid progress." science 384(6698):842-45. · DOI · Paper page
- Bostrom, Nick. 2024. Superintelligence : Dunod.
- Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M Fleming, Chris Frith, and Xu Ji. 2023. "Consciousness in artificial intelligence: insights from the science of consciousness." arXiv preprint arXiv:2308.08708 . · arXiv
- Choudhury, M, Anurag Dutta, and A Kumar De. 2023. "Data corroboration of the catastrophic chernobyl tragedy using arc-length estimate conjecture." Vertices: Duke's Undergraduate Research Journal 1(2).
- Christian, Brian. 2021. The alignment problem: How can machines learn human values? : Atlantic Books.
- Downing, Taylor. 2018. 1983: Reagan, Andropov, and a World on the Brink : Hachette UK.
- Greenblatt, Ryan, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, and David Duvenaud. 2024. "Alignment faking in large language models." arXiv preprint arXiv:2412.14093 . · arXiv
- Guha, Neil, Christie M Lawrence, Lindsey A Gailmard, Kit T Rodolfa, Faiz Surani, Rishi Bommasani, Inioluwa Deborah Raji, Mariano-Florentino Cuéllar, Colleen Honigsberg, and Percy Liang. 2024. "Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licensing, and auditing." Geo. Wash. L. Rev. 92:1473.
- Hellemans, Marc. 2009. The safety relief valve handbook: Design and use of process safety valves to ASME and International Codes and Standards : Elsevier.
- Hill, Claire A, and Erin O'Hara O'Connor. 2006. "A cognitive theory of trust." Washington University Law Review 84(7):1717-96.
- Hoffman, David. 1999. "I had a funny feeling in my gut." Washington Post 10:A19. Hood, Christopher, Henry Rothstein, and Robert Baldwin. 2001.
- Ingram, Scott. 2005. The Chernobyl nuclear disaster : Infobase Publishing.
- Kaminski, Margot E. 2023. "Regulating the Risks of AI." BUL Rev. 103:1347.
- Maggetti, Martino, Yannis Papadopolos, and Edoardo Guaschino. 2023. "Trust and Distrust in Regulatory Governance. White paper integrating the results of the TiGRE project and developing scenarios and recommendations to maintain and restore trust." Pp. 1-12. Lausanne: University of Lausanne.
- Malinauskas, AP. 1987. "The Chernobyl accident: Causes and consequences." Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States).
- Marcus, Gary, and Ernest Davis. 2019. Rebooting AI: Building artificial intelligence we can trust : Vintage.
- McDermott, Drew. 2007. "Artificial intelligence and consciousness." The Cambridge handbook of consciousness :117-50.
- Meinke, Alexander, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. 2024. "Frontier models are capable of in-context scheming." arXiv preprint arXiv:2412.04984 . · arXiv
- Morgan, Craig A. 1985. "The Downing of Korean Air Lines Flight 007." Yale J. Int'l L. 11:231.
- Ord, Toby. 2020. The precipice: Existential risk and the future of humanity : Hachette UK.
- Santos, Omar, and Petar Radanliev. 2024. Beyond the Algorithm: AI, Security, Privacy, and Ethics : Addison-Wesley Professional.
- Scharowski, Nicolas, Sebastian AC Perrig, Lena Fanya Aeschbach, Nick von Felten, Klaus Opwis, Philipp Wintersberger, and Florian Brühlmann. 2024. "To trust or distrust trust measures: Validating questionnaires for trust in ai." arXiv preprint arXiv:2403.00582 . · arXiv
- Singer P (1972) Famine, affluence, and morality. Philos Public Aff 1:229–243
- Six, Frédérique, and Koen Verhoest (Eds.). 2017. Trust in regulatory regimes : Edward Elgar Publishing.
- Sunstein, Cass R, and Jared H Gaffe. 2024. "An Anatomy of Algorithm Aversion." Colum. Sci. & Tech. L. Rev. 26:290.
- Sztompka, Piotr. 1999. Trust: A sociological theory : Cambridge university press.
- Tetlock, P., and A. Belkin. 1996. Counterfactual thought experiments in world politics: Logical, methodological, and psychological perspectives : Princeton University Press.
- Wang, Yi-Fan, Yu-Che Chen, Shih-Yi Chien, and Pin-Jen Wang. 2024. "Citizens' trust in AI-enabled government systems." Information Polity 29(3):293-312.
- Yablokov, Alexey V, Vassily B Nesterenko, Alexey V Nesterenko, and Janette D ShermanNevinger. 2010. Chernobyl: Consequences of the Catastrophe for People and the Environment : John Wiley & Sons.
- Bresinsky M, Hager E & Xanthos G (2024). ChatGPT for futures: how large language models can support the development of future scenarios using the Cone of Plausibility. CCEW SYMPOSIUM 2024: Predictive Synergies: Crisis early earning & foresight, 19. und 20. September 2024, München
- Wang L (2021). Autonomy vs. artifi cial intelligence : studies on healthcare work and analytics. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/146910
Cited by
Hostile interaction design: AI, governance, and the quest for human oversight
Foth, M. (2026) · AI & SOCIETY · doi:10.1007/s00146-026-03065-y · Cited by 1
“What matters is how authority, responsibility, and decision-making are calibrated across human and machine actors under conditions of uncertainty and risk (Maggetti 2026).”
Üstündağ, B., Foth, M., & Seevinck, J. (2026) · AI & SOCIETY · doi:10.1007/s00146-026-03108-4
“…a safeguard must itself be treated with caution: rather than a straightforward solu-tion, it represents a complex governance challenge shaped by institutional design, competing logics of trust and distrust, and the distributed nature of control in AI systems (Laux 2024; Maggetti 2026; Foth 2026).”
Metadata
- Authors
- Maggetti, M.
- Affiliations
- University of Lausanne (CH)
- Corresponding author
- Maggetti, Martino
- ORCID
- Maggetti, Martino
- Venue
- AI & SOCIETY
- Published
- 2026-03-17
- Volume
- 41
- Issue
- 5
- Pages
- 5009-5026
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- PMC
- PMC13690886
- OpenAlex
- W7138063264
- Cited by
- 2
- References
- 83
- Citations by year
- 2026: 2
- FWCI
- 11.35
- Type
- journal article
- Language
- en
- Update notices
- correction (2026-05-07)
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Open access funding provided by University of Lausanne Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (204521)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Computer Networks and Communications › Access Control and Trust
- Keywords
- AI regulation, trust in AI, distrust, mutual trust, AI agency, human-AI relationship, AI governance, democratic governance, trustworthiness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.