CoChat

Current cases of AI misalignment and their implications for future risks

Open access
Dung, Leonard · 2023 · Synthese
Cited by 56

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Abstract

How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time,misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11229-023-04367-0

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 35

  1. Armstrong, S., Bostrom, N., & Shulman, C. (2016). Racing to the precipice: A model of artificial intelligence development. AI & SOCIETY, 31(2), 201–206. https://doi.org/10.1007/s00146-015-0590-y. · DOI
  2. Arrhenius, G., Bykvist, K., Campbell, T., & Finneron-Burns, E. (Eds.). (2022). The Oxford Handbook of Population Ethics (1st ed.). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190907686.001.0001. · doi:10.1093/oxfordhb/9780190907686.001.0001
  3. Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens (arXiv:2303.08112). arXiv. http://arxiv.org/abs/2303.08112. · arXiv
  4. Bostrom, N. (2014). Superintelligence. Paths, Dangers, strategies. Oxford University Press.
  5. Bostrom, N., Dafoe, A., & Flynn, C. (2020). Public Policy and Superintelligent AI: A Vector Field Approach. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 292–326). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0011. · DOI
  6. Buckner, C. J. (2021). Black Boxes, or unflattering mirrors? Comparative Bias in the Science of Machine Behavior. The British Journal for the Philosophy of Science. https://doi.org/10.1086/714960. · DOI
  7. Burr, C., Cristianini, N., & Ladyman, J. (2018). An analysis of the Interaction between Intelligent Software Agents and human users. Minds and Machines, 28(4), 735–774. https://doi.org/10.1007/s11023-018-9479-0. · DOI · Paper page
  8. Carnap, R. (1950). Logical foundations of probability. Chicago University Press.
  9. Cave, S., & ÓhÉigeartaigh, S. S. (2018). An AI Race for Strategic Advantage: Rhetoric and Risks. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 36–40. https://doi.org/10.1145/3278721.3278780. · DOI · Paper page
  10. Dennett, D. C. (1987). The intentional stance.. MIT Press. · DOI · Paper page
  11. Dennett, D. C. (1991). Real patterns. Journal of Philosophy, 88(1), 27–51. · DOI · Paper page
  12. Dung, L. (2023a). How to deal with risks of AI suffering. Inquiry, 1–29.https://doi.org/10.1080/0020174X.2023.2238287 · DOI · Paper page
  13. Dung, L. (2023b). The argument for near-term human disempowerment through AI. https://philpapers.org/rec/DUNTAF-3. · DOI · Paper page
  14. Friederich, S. (2023). Symbiosis, not alignment, as the goal for liberal democracies in the transition to artificial general intelligence. AI and Ethics. https://doi.org/10.1007/s43681-023-00268-7. · DOI
  15. Glock, H. J. (2019). Agency, intelligence and reasons in animals. Philosophy, 94(4), 645–671. https://doi.org/10.1017/S0031819119000275. · DOI
  16. Greaves, H. (2017). Population Axiology. Philosophy Compass, 12(11), https://doi.org/10.1111/phc3.12442. · DOI
  17. Häggström, O. (2021). AI, orthogonality and the Muller-Cannon instrumental vs general intelligence distinction (arXiv:2109.07911). arXiv. https://doi.org/10.48550/arXiv.2109.07911. · DOI
  18. Hubinger, E. (2020). An overview of 11 proposals for building safe advanced AI (arXiv:2012.07532). arXiv. https://doi.org/10.48550/arXiv.2012.07532. · DOI
  19. Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., & Krueger, D. (2023). Goal Misgeneralization in Deep Reinforcement Learning (arXiv:2105.14111). arXiv. https://doi.org/10.48550/arXiv.2105.14111. · DOI
  20. Müller, V. C., & Cannon, M. (2022). Existential risk from AI and orthogonality: Can we have it both ways? Ratio, 35(1), 25–36. https://doi.org/10.1111/rati.12320. · DOI
  21. Ngo, R., Chan, L., & Mindermann, S. (2022). The alignment problem from a deep learning perspective (arXiv:2209.00626). arXiv. http://arxiv.org/abs/2209.00626. · arXiv
  22. OpenAI (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774. · DOI · Paper page
  23. Petersen, S. (2017). Superintelligence as Superethical. In P. Lin, K. Abney, & R. Jenkins (Eds.), Robot Ethics 2. 0: New Challenges in Philosophy, Law, and Society (pp. 322–337). Oxford University Press. https://philarchive.org/rec/PETSAS-12. · DOI
  24. Petersen, S. (2020). Machines learning values. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 413–436). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0015. · DOI
  25. Railton, P. (2020). Ethical Learning, Natural and Artificial. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 45–78). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0002. · DOI
  26. Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.
  27. Saad, B., & Bradley, A. (2022). Digital suffering: Why it’s a problem and how to prevent it. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–36. https://doi.org/10.1080/0020174X.2022.2144442. · DOI
  28. Shanahan, M. (2023). Talking About Large Language Models (arXiv:2212.03551). arXiv. https://doi.org/10.48550/arXiv.2212.03551. · arXiv · doi:10.48550/arxiv.2212.03551 · Paper page
  29. Skalse, J., Howe, N. H. R., Krasheninnikov, D., & Krueger, D. (2022). Defining and Characterizing Reward Hacking (arXiv:2209.13085). arXiv. https://doi.org/10.48550/arXiv.2209.13085. · DOI
  30. Sparrow, R. (2023). Friendly AI will still be our master. Or, why we should not want to be the pets of super-intelligent computers. AI & SOCIETY. https://doi.org/10.1007/s00146-023-01698-x. · DOI
  31. Taylor, J., Yudkowsky, E., LaVictoire, P., & Critch, A. (2020). Alignment for Advanced Machine Learning Systems. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 342–382). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0013. · DOI · Paper page
  32. Toromanoff, M., Wirbel, E., & Moutarde, F. (2019). Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field (arXiv:1908.04683). arXiv. https://doi.org/10.48550/arXiv.1908.04683. · DOI
  33. Torres, P. (2019). Existential risks: A philosophical analysis. Inquiry : A Journal of Medical Care Organization, Provision and Financing, 0(0), 1–26. https://doi.org/10.1080/0020174X.2019.1658626. · DOI
  34. Vold, K., & Harris, D. R. (2021). How Does Artificial Intelligence Pose an Existential Risk? In C. Véliz (Ed.), The Oxford Handbook of Digital Ethics (p. 0). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780198857815.013.36. · DOI
  35. Wallach, W., & Vallor, S. (2020). Moral Machines: From Value Alignment to Embodied Virtue. In S. M. Liao (Ed.), Ethics of Artificial Intelligence (pp. 383–412). Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0014. · DOI

Cited by

See all 56

Metadata

Authors
Dung, L.
Venue
Synthese
Published
2023-10-26
Volume
202
Issue
5
ISSN
  • 0039-7857 (ISSN-L)
  • 1573-0964
OpenAlex
W4388198642
Cited by
56
References
24
FWCI
13.67
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
Friedrich-Alexander-Universität Erlangen-Nürnberg

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Current cases of AI misalignment and their implications for future risks. https://app.cochat.ai/papers/10.1007/s11229-023-04367-0
The paper
Dung, L. (2023). Current cases of AI misalignment and their implications for future risks. Synthese. https://doi.org/10.1007/s11229-023-04367-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.