Current cases of AI misalignment and their implications for future risks
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time,misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 2×
“1 For this reason, we have to reflect on which values, or whose values, AI systems should ultimately be aligned with (Gabriel, 2020; Wallach & Vallor, 2020 ).”
Existential Risk Prevention as Global Priority
Bostrom, N. (2013) · Global Policy · doi:10.1111/1758-5899.12002 · Cited by 588 · cited 1×
“In the philosophical literature, it is common to define an existential risk as the risk of an outcome which permanently destroys the potential of humanity (Bostrom, 2013; Ord, 2020; Torres, 2019; Vold & Harris, 2021 ).”
Auditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580
Viewpoint: when will AI exceed human performance? evidence from AI experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11222 · Cited by 579
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1×
“The subsequent training serves to make the outputs of the system more “helpful, honest, and harmless” (Bai et al., 2022; Ouyang et al., 2022 ).”
BackgroundEmergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424
When Will AI Exceed Human Performance? Evidence from AI Experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2017) · arXiv · doi:10.48550/arxiv.1705.08807 · Cited by 285 · cited 1×
“A survey of expert AI researchers on the question, conducted in 2016, finds a mean forecast of the people surveyed of a 50% chance that “unaided machines can accomplish every task better and more cheaply than human workers” in 2061 (Grace et al., 2018 ).”
BackgroundChatGPT is not all you need. A state-of-the-art review of large generative AI models
Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023) · arXiv · doi:10.48550/arxiv.2301.04655 · Cited by 245
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196 · cited 1×
“The notion of goal, and the related notion of being an agent, is notoriously multi-facetted (Butlin, 2023 ).”
BackgroundReward learning from human preferences and demonstrations in Atari
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1811.06521 · Cited by 117
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · cited 2×
“Hence, it seems that the systems However, this does not perfectly correlate with producing helpful, honest and harm-less text which is what its designers want ChatGPT to do (Casper et al., 2023 ).”
BackgroundDiscovering Latent Knowledge in Language Models Without Supervision
Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2212.03827 · Cited by 55 · cited 1×
“14 A fourth argument that LLMs exhibit alignment, not just capability failures, is provided by studies along the lines of Burns et al. (2022 ), Halawi et al. ( 2023 ) and Belrose et al. ( 2023 ).”
BackgroundChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 2×
“…that performance of GPT-3 on reasoning tasks and math improves when the input is prefaced by the phrase “let’s think this through step by step” (Suzgun et al., 2022 ) or that ChatGPT’s texts can be improved by telling it that it is an excellent writer or an expert on the subject matter at hand.”
BackgroundReinforcement learning and artificial agency
Butlin, P. (2024) · Mind & Language · doi:10.1111/mila.12458 · Cited by 40
Is Power-Seeking AI an Existential Risk?
Carlsmith, J. (2022) · arXiv · doi:10.48550/arxiv.2206.13353 · Cited by 37
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Pan, A., Bhatia, K., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2201.03544 · Cited by 25
Overthinking the Truth: Understanding how Language Models Process False Demonstrations
Halawi, D., Denain, J.-S., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09476 · Cited by 7
Research works printed but not linked: 35
- Armstrong, S., Bostrom, N., & Shulman, C. (2016). Racing to the precipice: A model of artificial intelligence development. AI & SOCIETY, 31(2), 201–206. https://doi.org/10.1007/s00146-015-0590-y. · DOI
- Arrhenius, G., Bykvist, K., Campbell, T., & Finneron-Burns, E. (Eds.). (2022). The Oxford Handbook of Population Ethics (1st ed.). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190907686.001.0001. · doi:10.1093/oxfordhb/9780190907686.001.0001
- Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens (arXiv:2303.08112). arXiv. http://arxiv.org/abs/2303.08112. · arXiv
- Bostrom, N. (2014). Superintelligence. Paths, Dangers, strategies. Oxford University Press.
- Bostrom, N., Dafoe, A., & Flynn, C. (2020). Public Policy and Superintelligent AI: A Vector Field Approach. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 292–326). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0011. · DOI
- Buckner, C. J. (2021). Black Boxes, or unflattering mirrors? Comparative Bias in the Science of Machine Behavior. The British Journal for the Philosophy of Science. https://doi.org/10.1086/714960. · DOI
- Burr, C., Cristianini, N., & Ladyman, J. (2018). An analysis of the Interaction between Intelligent Software Agents and human users. Minds and Machines, 28(4), 735–774. https://doi.org/10.1007/s11023-018-9479-0. · DOI · Paper page
- Carnap, R. (1950). Logical foundations of probability. Chicago University Press.
- Cave, S., & ÓhÉigeartaigh, S. S. (2018). An AI Race for Strategic Advantage: Rhetoric and Risks. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 36–40. https://doi.org/10.1145/3278721.3278780. · DOI · Paper page
- Dennett, D. C. (1987). The intentional stance.. MIT Press. · DOI · Paper page
- Dennett, D. C. (1991). Real patterns. Journal of Philosophy, 88(1), 27–51. · DOI · Paper page
- Dung, L. (2023a). How to deal with risks of AI suffering. Inquiry, 1–29.https://doi.org/10.1080/0020174X.2023.2238287 · DOI · Paper page
- Dung, L. (2023b). The argument for near-term human disempowerment through AI. https://philpapers.org/rec/DUNTAF-3. · DOI · Paper page
- Friederich, S. (2023). Symbiosis, not alignment, as the goal for liberal democracies in the transition to artificial general intelligence. AI and Ethics. https://doi.org/10.1007/s43681-023-00268-7. · DOI
- Glock, H. J. (2019). Agency, intelligence and reasons in animals. Philosophy, 94(4), 645–671. https://doi.org/10.1017/S0031819119000275. · DOI
- Greaves, H. (2017). Population Axiology. Philosophy Compass, 12(11), https://doi.org/10.1111/phc3.12442. · DOI
- Häggström, O. (2021). AI, orthogonality and the Muller-Cannon instrumental vs general intelligence distinction (arXiv:2109.07911). arXiv. https://doi.org/10.48550/arXiv.2109.07911. · DOI
- Hubinger, E. (2020). An overview of 11 proposals for building safe advanced AI (arXiv:2012.07532). arXiv. https://doi.org/10.48550/arXiv.2012.07532. · DOI
- Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., & Krueger, D. (2023). Goal Misgeneralization in Deep Reinforcement Learning (arXiv:2105.14111). arXiv. https://doi.org/10.48550/arXiv.2105.14111. · DOI
- Müller, V. C., & Cannon, M. (2022). Existential risk from AI and orthogonality: Can we have it both ways? Ratio, 35(1), 25–36. https://doi.org/10.1111/rati.12320. · DOI
- Ngo, R., Chan, L., & Mindermann, S. (2022). The alignment problem from a deep learning perspective (arXiv:2209.00626). arXiv. http://arxiv.org/abs/2209.00626. · arXiv
- OpenAI (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774. · DOI · Paper page
- Petersen, S. (2017). Superintelligence as Superethical. In P. Lin, K. Abney, & R. Jenkins (Eds.), Robot Ethics 2. 0: New Challenges in Philosophy, Law, and Society (pp. 322–337). Oxford University Press. https://philarchive.org/rec/PETSAS-12. · DOI
- Petersen, S. (2020). Machines learning values. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 413–436). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0015. · DOI
- Railton, P. (2020). Ethical Learning, Natural and Artificial. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 45–78). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0002. · DOI
- Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.
- Saad, B., & Bradley, A. (2022). Digital suffering: Why it’s a problem and how to prevent it. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–36. https://doi.org/10.1080/0020174X.2022.2144442. · DOI
- Shanahan, M. (2023). Talking About Large Language Models (arXiv:2212.03551). arXiv. https://doi.org/10.48550/arXiv.2212.03551. · arXiv · doi:10.48550/arxiv.2212.03551 · Paper page
- Skalse, J., Howe, N. H. R., Krasheninnikov, D., & Krueger, D. (2022). Defining and Characterizing Reward Hacking (arXiv:2209.13085). arXiv. https://doi.org/10.48550/arXiv.2209.13085. · DOI
- Sparrow, R. (2023). Friendly AI will still be our master. Or, why we should not want to be the pets of super-intelligent computers. AI & SOCIETY. https://doi.org/10.1007/s00146-023-01698-x. · DOI
- Taylor, J., Yudkowsky, E., LaVictoire, P., & Critch, A. (2020). Alignment for Advanced Machine Learning Systems. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 342–382). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0013. · DOI · Paper page
- Toromanoff, M., Wirbel, E., & Moutarde, F. (2019). Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field (arXiv:1908.04683). arXiv. https://doi.org/10.48550/arXiv.1908.04683. · DOI
- Torres, P. (2019). Existential risks: A philosophical analysis. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–26. https://doi.org/10.1080/0020174X.2019.1658626. · DOI
- Vold, K., & Harris, D. R. (2021). How Does Artificial Intelligence Pose an Existential Risk? In C. Véliz (Ed.), The Oxford Handbook of Digital Ethics (p. 0). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780198857815.013.36. · DOI
- Wallach, W., & Vallor, S. (2020). Moral Machines: From Value Alignment to Embodied Virtue. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 383–412). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0014. · DOI
Cited by
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
“A primary concern is the emergence of human-level or superhuman generative models, commonly referred to as AGI, and their potential existential or catastrophic risks to humanity (Ben-gio et al., 2023; Dung, 2023b; Hendrycks et al., 2023; Koessler & Schuett, 2023).”
Understanding artificial agency
Dung, L. (2025) · The Philosophical Quarterly · doi:10.1093/pq/pqae010 · Cited by 69
“A significant number of authors has expressed concern that advanced AI poses an existential risk to humanity (e.g. Bostrom 2014; Carlsmith 2022; Dung 2023a; Dung 2023b; Ngo, Chan, and Mindermann 2022; Russell 2019).”
BackgroundSelf-citationThe bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
Metadata
- Authors
- Dung, L.
- Venue
- Synthese
- Published
- 2023-10-26
- Volume
- 202
- Issue
- 5
- ISSN
- 0039-7857 (ISSN-L)
- 1573-0964
- OpenAlex
- W4388198642
- Cited by
- 56
- References
- 24
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Friedrich-Alexander-Universität Erlangen-Nürnberg
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.