The argument for near-term human disempowerment through AI
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“…It has been shown that language model performance improves smoothly as the scale of the model increases, where performance has a power–law relationship with each individual factor (model size, size of the training dataset or compute) when not bot-tlenecked by the other two (Kaplan et al. 2020).”
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1×
“The canonical answer in the literature is the instrumental convergence thesis (Bostrom 2012, 2014).”
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 1×
“The underlying rationale is that many, or all, misaligned AI models capable of disempowering humanity will have the goal of disempowering humanity.”
Superintelligence: Paths, Dangers, Strategies
Mulgan, T. (2015) · The Philosophical Quarterly · doi:10.1093/pq/pqv034 · Cited by 878
Deep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853
Existential Risk Prevention as Global Priority
Bostrom, N. (2013) · Global Policy · doi:10.1111/1758-5899.12002 · Cited by 588
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1×
“Remarkable capacities for compositionality (Press et al. 2022; Yao et al. 2021) and domain-general reasoning (Bubeck et al. 2023; OpenAI 2023) are present in modern LLMs.”
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287
The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
Bostrom, N. (2012) · Minds and Machines · doi:10.1007/s11023-012-9281-3 · Cited by 254 · cited 1×
“The canonical answer in the literature is the instrumental convergence thesis (Bostrom 2012, 2014).”
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · cited 1×
“Alternatively, a proponent of the argument developed here may argue that there are incentives to intentionally create agentic AI systems (see, e.g., the creation of so-called language agents) (Chan et al. 2023). of data humans have access to during ontogeny (Buckner 2021).”
Understanding artificial agency
Dung, L. (2025) · The Philosophical Quarterly · doi:10.1093/pq/pqae010 · Cited by 69
Self-citationEmergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61
Discovering Latent Knowledge in Language Models Without Supervision
Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2212.03827 · Cited by 55 · cited 1×
“Other AI paradigms also benefit from availability of more compute and better hardware.”
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 1×
“Nevertheless, in the last five years, performance in virtually all interesting benchmark tasks has improved massively with increased scale (OpenAI 2023; Suzgun et al. 2022).”
Is Power-Seeking AI an Existential Risk?
Carlsmith, J. (2022) · arXiv · doi:10.48550/arxiv.2206.13353 · Cited by 37
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Pan, A., Bhatia, K., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2201.03544 · Cited by 25
Collaborating with language models for embodied reasoning
Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., & Fergus, R. (2023) · arXiv · doi:10.48550/arxiv.2302.00763 · Cited by 14
Cited by
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
Understanding artificial agency
Dung, L. (2025) · The Philosophical Quarterly · doi:10.1093/pq/pqae010 · Cited by 69
Self-citationAnthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713329 · Cited by 24
“…fail to prepare for these new forms of human-AI interaction, we could face “disempowerment,” absent-mindedly giving up control to evolutionary or artificial forces [37, 41, 45, 57, 94, 146], and this transition may be particularly difficult to preempt if it is “accumulative” [86] or “gradual” [94].”
Anthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2024) · arXiv · doi:10.48550/arxiv.2407.08867 · Cited by 24
Examining popular arguments against AI existential risk: a philosophical analysis
Swoboda, T., Uuk, R., Lauwaert, L., Rebera, A. P., Oimann, A.-K., Chomanski, B., & Prunkl, C. (2026) · Ethics and Information Technology · doi:10.1007/s10676-025-09881-y · Cited by 1
“For one thing, if superintelligent AI does pose existential risks—an issue that the Argument from Human Frailty does not address—humanity likely gets only one chance to get the response right: get it wrong, and disaster is upon us (Bostrom, 2014; Dung, 2024; Ord, 2020).”
Metadata
- Authors
- Dung, L.
- Venue
- AI & SOCIETY
- Published
- 2024-04-14
- Volume
- 40
- Issue
- 3
- Pages
- 1195-1208
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W4394793541
- Cited by
- 27
- References
- 26
- FWCI
- 10.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.