A model of pathways to artificial superintelligence catastrophe for risk and decision analysis
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
An artificial superintelligence (ASI) is an artificial intelligence that is significantly more intelligent than humans in all respects. Whilst ASI does not currently exist, some scholars propose that it could be created sometime in the future, and furthermore that its creation could cause a severe global catastrophe, possibly even resulting in human extinction. Given the high stakes, it is important to analyze ASI risk and factor the risk into decisions related to ASI research and development. This paper presents a graphical model of major pathways to ASI catastrophe, focusing on ASI created via recursive self-improvement. The model uses the established risk and decision analysis modelling paradigms of fault trees and influence diagrams in order to depict combinations of events and conditions that could lead to AI catastrophe, as well as intervention options that could decrease risks. The events and conditions include select aspects of the ASI itself as well as the human process of ASI research, development and management. Model structure is derived from published literature on ASI risk. The model offers a foundation for rigorous quantitative evaluation and decision-making on the long-term risk of ASI catastrophe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Speculations Concerning the First Ultraintelligent Machine
Good, I. J. (1966) · Advances in Computers · doi:10.1016/s0065-2458(08)60418-0 · Cited by 703
Research Priorities for Robust and Beneficial Artificial Intelligence
Russell, S., Dewey, D., & Tegmark, M. (2015) · AI Magazine · doi:10.1609/aimag.v36i4.2577 · Cited by 637
Existential Risk Prevention as Global Priority
Bostrom, N. (2013) · Global Policy · doi:10.1111/1758-5899.12002 · Cited by 588
Artificial General Intelligence
(2007) · Cognitive Technologies · doi:10.1007/978-3-540-68677-4 · Cited by 473
Artificial Intelligence as a Positive and Negative Factor in Global Risk
Yudkowsky, E. (2008) · Global Catastrophic Risks · doi:10.1093/oso/9780198570509.003.0021 · Cited by 431
Omohundro, S. M. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-3 · Cited by 148
Thinking Inside the Box: Controlling and Using an Oracle AI
Armstrong, S., Sandberg, A., & Bostrom, N. (2012) · Minds and Machines · doi:10.1007/s11023-012-9282-2 · Cited by 144
Responses to catastrophic AGI risk: a survey
Sotala, K., & Yampolskiy, R. V. (2015) · Physica Scripta · doi:10.1088/0031-8949/90/1/018001 · Cited by 121 · cited 8×
“However, supporters of requiring mathematical proofs have argued that the belief that proofs are unnecessary might give AI designers a dangerous misperception that an AI’s goals would remain safe outside its training environment (Sotala & Yampolskiy 2015 pp. 21-23).”
BackgroundInfluentialAnalyzing and reducing the risks of inadvertent nuclear war between the United States and Russia
Barrett, A. M., Baum, S. D., & Hostetler, K. (2013) · Science & Global Security · doi:10.1080/08929882.2013.798984 · Cited by 81
Self-citation
Cited by
Grassini, S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1191628 · Cited by 330
“The AIAS items were developed based on established theoretical frameworks and empirical research in technology acceptance, AI, and risk perception (Davis, 1989; Venkatesh et al., 2003; Barrett and Baum, 2017).”
BackgroundMethodA Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy
Baum, S. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3070741 · Cited by 93
Self-citationGrassini, S., & Ree, A. S. (2023) · Proceedings of the European Conference on Cognitive Ergonomics 2023 · doi:10.1145/3605655.3605669 · Cited by 50
“intelligent AI [48] and the risk perception related to technology [49].”
BackgroundA Reasoning and Value Alignment Test to Assess Advanced GPT Reasoning
Mcintosh, T. R., Liu, T., Susnjak, T., Watters, P., & Halgamuge, M. N. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3670691 · Cited by 42
Metadata
- Authors
- Barrett, A. M., & Baum, S. D.
- Venue
- Journal of Experimental & Theoretical Artificial Intelligence
- Publisher
- Informa UK Limited
- Published
- 2016-05-23
- Volume
- 29
- Issue
- 2
- Pages
- 397-414
- ISSN
- 0952-813X (ISSN-L)
- 1362-3079
- OpenAlex
- W2399666893
- Cited by
- 66
- References
- 9
- FWCI
- 11.20
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. https://app.cochat.ai/papers/10.1080/0952813x.2016.1186228
- The paper
- Barrett, A. M., & Baum, S. D. (2017). A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. Journal of Experimental & Theoretical Artificial Intelligence. https://doi.org/10.1080/0952813x.2016.1186228
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.