Current cases of AI misalignment and their implications for future risks
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time,misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 2×
“1 For this reason, we have to reflect on which values, or whose values, AI systems should ultimately be aligned with (Gabriel, 2020; Wallach & Vallor, 2020 ).”
Existential Risk Prevention as Global Priority
Bostrom, N. (2013) · Global Policy · doi:10.1111/1758-5899.12002 · Cited by 588 · cited 1×
“In the philosophical literature, it is common to define an existential risk as the risk of an outcome which permanently destroys the potential of humanity (Bostrom, 2013; Ord, 2020; Torres, 2019; Vold & Harris, 2021 ).”
Auditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580
Research works printed but not linked: 35
- Armstrong, S., Bostrom, N., & Shulman, C. (2016). Racing to the precipice: A model of artificial intelligence development. AI & SOCIETY, 31(2), 201–206. https://doi.org/10.1007/s00146-015-0590-y. · DOI
- Arrhenius, G., Bykvist, K., Campbell, T., & Finneron-Burns, E. (Eds.). (2022). The Oxford Handbook of Population Ethics (1st ed.). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190907686.001.0001. · doi:10.1093/oxfordhb/9780190907686.001.0001
- Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens (arXiv:2303.08112). arXiv. http://arxiv.org/abs/2303.08112. · arXiv
- Bostrom, N. (2014). Superintelligence. Paths, Dangers, strategies. Oxford University Press.
- Bostrom, N., Dafoe, A., & Flynn, C. (2020). Public Policy and Superintelligent AI: A Vector Field Approach. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 292–326). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0011. · DOI
- Buckner, C. J. (2021). Black Boxes, or unflattering mirrors? Comparative Bias in the Science of Machine Behavior. The British Journal for the Philosophy of Science. https://doi.org/10.1086/714960. · DOI
- Burr, C., Cristianini, N., & Ladyman, J. (2018). An analysis of the Interaction between Intelligent Software Agents and human users. Minds and Machines, 28(4), 735–774. https://doi.org/10.1007/s11023-018-9479-0. · DOI · Paper page
- Carnap, R. (1950). Logical foundations of probability. Chicago University Press.
- Cave, S., & ÓhÉigeartaigh, S. S. (2018). An AI Race for Strategic Advantage: Rhetoric and Risks. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 36–40. https://doi.org/10.1145/3278721.3278780. · DOI · Paper page
- Dennett, D. C. (1987). The intentional stance.. MIT Press. · DOI · Paper page
- Dennett, D. C. (1991). Real patterns. Journal of Philosophy, 88(1), 27–51. · DOI · Paper page
- Dung, L. (2023a). How to deal with risks of AI suffering. Inquiry, 1–29.https://doi.org/10.1080/0020174X.2023.2238287 · DOI · Paper page
- Dung, L. (2023b). The argument for near-term human disempowerment through AI. https://philpapers.org/rec/DUNTAF-3. · DOI · Paper page
- Friederich, S. (2023). Symbiosis, not alignment, as the goal for liberal democracies in the transition to artificial general intelligence. AI and Ethics. https://doi.org/10.1007/s43681-023-00268-7. · DOI
- Glock, H. J. (2019). Agency, intelligence and reasons in animals. Philosophy, 94(4), 645–671. https://doi.org/10.1017/S0031819119000275. · DOI
- Greaves, H. (2017). Population Axiology. Philosophy Compass, 12(11), https://doi.org/10.1111/phc3.12442. · DOI
- Häggström, O. (2021). AI, orthogonality and the Muller-Cannon instrumental vs general intelligence distinction (arXiv:2109.07911). arXiv. https://doi.org/10.48550/arXiv.2109.07911. · DOI
- Hubinger, E. (2020). An overview of 11 proposals for building safe advanced AI (arXiv:2012.07532). arXiv. https://doi.org/10.48550/arXiv.2012.07532. · DOI
- Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., & Krueger, D. (2023). Goal Misgeneralization in Deep Reinforcement Learning (arXiv:2105.14111). arXiv. https://doi.org/10.48550/arXiv.2105.14111. · DOI
- Müller, V. C., & Cannon, M. (2022). Existential risk from AI and orthogonality: Can we have it both ways? Ratio, 35(1), 25–36. https://doi.org/10.1111/rati.12320. · DOI
- Ngo, R., Chan, L., & Mindermann, S. (2022). The alignment problem from a deep learning perspective (arXiv:2209.00626). arXiv. http://arxiv.org/abs/2209.00626. · arXiv
- OpenAI (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774. · DOI · Paper page
- Petersen, S. (2017). Superintelligence as Superethical. In P. Lin, K. Abney, & R. Jenkins (Eds.), Robot Ethics 2. 0: New Challenges in Philosophy, Law, and Society (pp. 322–337). Oxford University Press. https://philarchive.org/rec/PETSAS-12. · DOI
- Petersen, S. (2020). Machines learning values. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 413–436). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0015. · DOI
- Railton, P. (2020). Ethical Learning, Natural and Artificial. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 45–78). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0002. · DOI
- Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.
- Saad, B., & Bradley, A. (2022). Digital suffering: Why it’s a problem and how to prevent it. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–36. https://doi.org/10.1080/0020174X.2022.2144442. · DOI
- Shanahan, M. (2023). Talking About Large Language Models (arXiv:2212.03551). arXiv. https://doi.org/10.48550/arXiv.2212.03551. · arXiv · doi:10.48550/arxiv.2212.03551 · Paper page
- Skalse, J., Howe, N. H. R., Krasheninnikov, D., & Krueger, D. (2022). Defining and Characterizing Reward Hacking (arXiv:2209.13085). arXiv. https://doi.org/10.48550/arXiv.2209.13085. · DOI
- Sparrow, R. (2023). Friendly AI will still be our master. Or, why we should not want to be the pets of super-intelligent computers. AI & SOCIETY. https://doi.org/10.1007/s00146-023-01698-x. · DOI
- Taylor, J., Yudkowsky, E., LaVictoire, P., & Critch, A. (2020). Alignment for Advanced Machine Learning Systems. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 342–382). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0013. · DOI · Paper page
- Toromanoff, M., Wirbel, E., & Moutarde, F. (2019). Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field (arXiv:1908.04683). arXiv. https://doi.org/10.48550/arXiv.1908.04683. · DOI
- Torres, P. (2019). Existential risks: A philosophical analysis. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–26. https://doi.org/10.1080/0020174X.2019.1658626. · DOI
- Vold, K., & Harris, D. R. (2021). How Does Artificial Intelligence Pose an Existential Risk? In C. Véliz (Ed.), The Oxford Handbook of Digital Ethics (p. 0). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780198857815.013.36. · DOI
- Wallach, W., & Vallor, S. (2020). Moral Machines: From Value Alignment to Embodied Virtue. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 383–412). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0014. · DOI
Cited by
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
“A primary concern is the emergence of human-level or superhuman generative models, commonly referred to as AGI, and their potential existential or catastrophic risks to humanity (Ben-gio et al., 2023; Dung, 2023b; Hendrycks et al., 2023; Koessler & Schuett, 2023).”
Understanding artificial agency
Dung, L. (2025) · The Philosophical Quarterly · doi:10.1093/pq/pqae010 · Cited by 69
“A significant number of authors has expressed concern that advanced AI poses an existential risk to humanity (e.g. Bostrom 2014; Carlsmith 2022; Dung 2023a; Dung 2023b; Ngo, Chan, and Mindermann 2022; Russell 2019).”
BackgroundSelf-citationThe bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
Metadata
- Authors
- Dung, L.
- Venue
- Synthese
- Published
- 2023-10-26
- Volume
- 202
- Issue
- 5
- ISSN
- 0039-7857 (ISSN-L)
- 1573-0964
- OpenAlex
- W4388198642
- Cited by
- 56
- References
- 24
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Friedrich-Alexander-Universität Erlangen-Nürnberg
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Current cases of AI misalignment and their implications for future risks. https://app.cochat.ai/papers/10.1007/s11229-023-04367-0
- The paper
- Dung, L. (2023). Current cases of AI misalignment and their implications for future risks. Synthese. https://doi.org/10.1007/s11229-023-04367-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.