CoChat
Sign Up
Cited by 12Safety Systems Engineering in AutonomyEthics and Social Impacts of AIRisk and Safety Analysis

System Safety and Artificial Intelligence

Open access
Dobbe, Roel I. J.·2022·The Oxford Handbook of AI Governance
DOI 10.1093/oxfordhb/9780197579329.013.67
Open this paper in CoChat — free accountOpen PDF

Abstract

This chapter formulates seven lessons for preventing harm in artificial intelligence (AI) systems based on insights from the field of system safety for software-based automation in safety-critical domains. New applications of AI across societal domains and public organizations and infrastructures come with new hazards, which lead to new forms of harm, both grave and pernicious. The chapter addresses the lack of consensus for diagnosing and eliminating new AI system hazards. For decades, the field of system safety has dealt with accidents and harm in safety-critical systems governed by varying degrees of software-based automation and decision-making. This field embraces the core assumption of systems and control that AI systems cannot be safeguarded by technical design choices on the model or algorithm alone, instead requiring an end-to-end hazard analysis and design frame that includes the context of use, impacted stakeholders, and the formal and informal institutional environment in which the system operates. Safety and other values are then inherently socio-technical and emergent system properties that require design and control measures to instantiate these across the technical, social, and institutional components of a system. This chapter honors system safety pioneer Nancy Leveson, by situating her core lessons for today’s AI system safety challenges. For every lesson, concrete tools are offered for rethinking and reorganizing the safety management of AI systems, both in design and governance. This history tells us that effective AI safety management requires transdisciplinary approaches and a shared language that allows involvement of all levels of society.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/oxfordhb/9780197579329.013.67

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RLHF and RLAIF introduce further difficulties for properly understanding and designing the inner workings of LLMs in ways that improve accountability.

    “As such, RLHF and RLAIF introduce further difficulties for properly understanding and designing the inner workings of LLMs in ways that improve accountability (Dobbe, 2022).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Dobbe, 2022

    In Leveson Lesson 5 (page not provided), the paper describes LLMs as complex and difficult to equip with appropriate safety requirements. Leveson Lesson 7 discusses accountability gaps in AI systems. Neither passage attributes additional difficulties in understanding or designing LLMs to RLHF or RLAIF specifically. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • System safety embraces the core assumption that AI systems cannot be safeguarded by technical design choices centred on the model or algorithm alone, requiring instead a broad analysis and design frame including context of use, impacted stakeholders, and the institutional environment.

    “System safety embraces the core assumption that AI systems cannot be safeguarded by technical design choices centred on the model or algorithm alone, requiring instead a broad analysis and design frame that includes the context of use, impacted stakeholders, and the formal and informal institutional environment in which the system operates (Dobbe, 2022).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Dobbe, 2022

  • The need for system safety was recently acknowledged in the first International AI Safety Report, following earlier responses to the narrow focus on technical interventions.

    “The need for system safety was recently acknowledged in the first International AI Safety Report (Bengio et al., 2025), following earlier responses to the narrow focus on technical interventions (Raji & Dobbe, 2020; Dobbe, 2022).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Dobbe, 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 27 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Man´e, D. (2016). Concrete problems in AI safety. arXiv:1606.06565 [cs. · arXiv
  2. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610-623). FAccT ʼ21. New York, NY, USA. Association for Computing Machinery. Google Scholar Google Preview WorldCat
  3. Hu, L., & Kohler-Hausmann, I. (2020). Whatʼs sex got to do with fair machine learning? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (p. 513. Barcelona.
  4. Lum, K., & Isaac, W. (2016). To predict and serve? Significance 13(5), 14-19.
  5. Richardson, R., Schultz, J., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. New York University Law Review 94, 192-233.
  6. Whittaker, M. (2021). The steep cost of capture. Interactions 28(6), 50-55.

Cited by

Sorted by citations

Related

Authors
Dobbe, R. I. J.
Affiliations
Delft University of Technology (NL)
Corresponding author
Dobbe, Roel I. J.
ORCID
Dobbe, Roel I. J.
Venue
The Oxford Handbook of AI Governance
Publisher
Oxford University Press
Published
2022-10-20
Pages
441-458
DOI
10.1093/oxfordhb/9780197579329.013.67
OpenAlex
W4307697543
Cited by
12
References
27
Citations by year
2025: 4 · 2024: 6 · 2023: 1 · 2022: 1
FWCI
16.19
Type
book chapter
Language
en
Open access
Licence
public-domain
Locations
  • Oxford University Press eBooks (publishedVersion)
  • Research Repository (Delft University of Technology) (submittedVersion, public-domain)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Risk and Safety Analysis
Keywords
harm prevention, safety-critical systems, hazard analysis, AI governance, safety management, transdisciplinarity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review