The DOI matches its DataCite record; the references it cites are not checked yet.
Safety Cases: How to Justify the Safety of Advanced AI Systems
Open accessAbstract
As AI systems become more advanced, companies and regulators will make difficult decisions about whether it is safe to train and deploy them. To prepare for these decisions, we investigate how developers could make a 'safety case,' which is a structured rationale that AI systems are unlikely to cause a catastrophe. We propose a framework for organizing a safety case and discuss four categories of arguments to justify safety: total inability to cause a catastrophe, sufficiently strong control measures, trustworthiness despite capability to cause harm, and -- if AI systems become much more powerful -- deference to credible AI advisors. We evaluate concrete examples of arguments in each category and outline how arguments could be combined to justify that AI systems are safe to deploy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evidence of models' lack of capabilities in these domains can be used as part of a safety case that even misaligned models won't cause catastrophic harm if deployed.
“Evidence of models' lack of capabilities in these domains can then be used as part of a safety case (Clymer et al., 2024; Irving, 2024) that even misaligned models won't cause catastrophic harm if deployed.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Clymer et al., 2024
The paper's evaluations could be used to support inability- and control-style arguments according to Clymer et al.'s classification.
“Our evaluations could be used to support inability- and control-style arguments according to their classification.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Clymer et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists haven’t been fetched yet
Cited by 5
Research works printed but not linked: 16
- Armstrong, Stuart, Sandberg, Anders, & Bostrom, Nick. 2012. Thinking Inside the Box: Controlling and Using an Oracle AI. Minds and Machines, 22(4), 299-324.
- Bostrom, Nick. 2014. Superintelligence: Paths, Dangers, Strategies. 1st edn. USA: Oxford University Press, Inc.
- Chalmers, David J. 2010. The Singularity: A Philosophical Analysis. Journal of Consciousness Studies, 17(9-10), 9-10.
- Chen, Banghao, Zhang, Zhaofeng, Langrené, Nicolas, & Zhu, Shengxin. 2023. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review.
- Evans, Owain, Cotton-Barratt, Owen, Finnveden, Lukas, Bales, Adam, Balwit, Avital, Wills, Peter, Righetti, Luca, & Saunders, William. 2021. Truthful AI: Developing and governing AI that does not lie.
- Ferrara, Emilio. 2023. Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci, 6(1), 3.
- Grossman, Sanford J., & Hart, Oliver D. 1983. An Analysis of the Principal-Agent Problem. Econometrica, 51(1), 7-45.
- Haddon-Cave QC, Charles. 2009. An independent review into the broader issues surrounding the loss of the RAF Nimrod MR2 Aircraft XV230 in Afghanistan in 2006.
- Koessler, Leonie, & Schuett, Jonas. 2023. Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries.
- Kritzinger, Duane. 2017. 5 -Failure Modes and Effects Analysis. Pages 101-132 of: Kritzinger, Duane (ed), Aircraft System Safety. Woodhead Publishing.
- Leveson, Nancy. 2020. Are You Sure Your Software Will Not Kill Anyone? -Communications of the ACM. Acm.org, Feb.
- Morris, Allan, & Beling, Peter. 2001 (11). Space shuttle RTOS bayesian network. Pages 4D5/1 -4D5/13 vol.1 of: 20th Digital Avionics Systems Conference, vol. 1.
- Omohundro, Stephen M. 2008. The Basic AI Drives. Page 483-492 of: Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference. NLD: IOS Press.
- Sujan, Mark A., Habli, Ibrahim, Kelly, Tim P., Pozzi, Simone, & Johnson, Christopher W. 2016. Should healthcare providers do safety cases? Lessons from a cross-industry review of safety case practices. Safety Science, 84, 181-189.
- Urban, Caterina, & Miné, Antoine. 2021. A Review of Formal Methods applied to Machine Learning.
- Yampolskiy, Roman. 2012. Leakproofing the Singularity: Artificial Intelligence Confinement Problem. Jour- nal of Consciousness Studies, 19(1-2), 194-214.
Metadata
- Authors
- Clymer, J., Gabrieli, N., Krueger, D., & Larsen, T.
- Affiliations
- Columbia University (US)
- Harvard University (US)
- University of Cambridge (GB)
- ORCID
- Krueger, David
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.10462
- OpenAlex
- W4392933274
- Cited by
- 5
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- safety case, AI safety, catastrophic risk, control measures, trustworthiness, AI regulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
