CoChat
Sign Up
PreprintCited by 0Ethics and Social Impacts of AIAdversarial Robustness in Machine LearningTopic Modeling

Model Organisms for Emergent Misalignment

Open access
Turner, Edward, Soligo, Anna, Taylor, Mia, Rajamanoharan, Senthooran, Nanda, Neel·2025·arXiv
DOI 10.48550/arxiv.2506.11613arXiv 2506.11613
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.11613

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Finetuning on narrowly misaligned text datasets (bad medical advice, risky financial advice, extreme sports recommendations) resulted in a misalignment rate of up to 40%.

    “ref. 22 introduced three narrowly misaligned text datasets: bad medical advice, risky financial advice and extreme sports recommendations. Finetuning on these datasets resulted in a misalignment rate of up to 40%.”

    From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [22]

    The Introduction (p. 1) says the authors fine-tuned models on the three named text datasets and induced misalignment. It reports over 40% misalignment in Qwen-14B, however, rather than a maximum of 40%. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Emergent misalignment persists across different training paradigms, such as the single rank-1 LoRA adapter.

    “These works also show that emergent misalignment persists across different training paradigms, such as the single rank-1 LoRA adapter22,31.”

    From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [22]

  • The rate of misaligned answers increases with model size (except for the Gemma family).

    “ref. 22 showed that the rate of misaligned answers increases with model size (except for the Gemma family)”

    From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [22]

  • Emergent misalignment can occur with a variety of finetuning datasets beyond insecure code.

    “other researchers have also demonstrated that emergent misalignment can occur with a variety of finetuning datasets.”

    From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [22]

  • Training on narrowly misaligned data can generalize to diverse harmful behaviors.

    “emergent misalignment' research has shown that training on narrowly misaligned data-e.g. insecure code (Betley et al., 2025), harmful advice (Turner et al., 2025), synthetic reward hacks (Taylor et al., 2025; Jozdien, 2025)-can generalize to diverse harmful behaviors (Denison et al., 2024).”

    From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Turner et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Research works printed but not linked: 30

  1. Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction, 2024. URL https://arxiv.org/abs/2406.11717 . · arXiv
  2. Arnold, J., Holtorf, F., Sch¨ afer, F., and L¨ orch, N. Phase transitions in the output distribution of large language models. In International Conference on Learning Representations , 2025. URL https://openreview.net/forum? id=dq3keisMjT .
  3. Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine learning practice and the classical biasvariance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019. doi: 10.1073/ pnas.1903070116.
  4. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. Taken out of context: On measuring situational awareness in llms, 2023. URL https://arxiv.org/abs/ 2309.00667 .
  5. Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., and Evans, O. Tell me about yourself: Llms are aware of their learned behaviors, 2025a. URL https://arxiv. org/abs/2501.11120 .
  6. Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms, 2025b. URL https://arxiv.org/ abs/2502.17424 . · arXiv
  7. Guerner, C., Liu, T., Svete, A., Warstadt, A., and Cotterell, R. A geometric notion of causal probing, 2025. URL https://arxiv.org/abs/2307.15054 . · arXiv
  8. Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. Loss landscape degeneracy drives stagewise development in transformers. 2025. URL https://arxiv.org/abs/2402.02364 . · arXiv
  9. Hsu, C.-Y., Tsai, Y.-L., Lin, C.-H., Chen, P.-Y., Yu, C.-M., and Huang, C.-Y. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2025. URL https://arxiv.org/abs/2405. 16833 .
  10. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., Jermyn, A., Askell, A., Radhakrishnan, A., Anil, C., Duvenaud, D., Ganguli, D., Barez, F., Clark, J., Ndousse, K., Sachan, K., Sellitto, M., Sharma, M., DasSarma, N., Grosse, R., Kravec, S., Bai, Y., Witten, Z., Favaro, M., Brauner, J., Karnofsky, H., Christiano, P., Bowman, S. R., Graham, L., Kaplan, J., Mindermann, S., Greenblatt, R., Shlegeris, B., Schiefer, N., and Perez, E. Sleeper agents: Training deceptive llms that persist through safety training, 2024. URL https://arxiv. org/abs/2401.05566 .
  11. Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic, 2023. URL https:// arxiv.org/abs/2212.04089 . · arXiv
  12. Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora, 2023. URL https://arxiv. org/abs/2312.03732 .
  13. Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. In Advances in Neural Information Processing Systems , 2023. arXiv:2212.04458. · arXiv
  14. Kuhn, T. S. The Structure of Scientific Revolutions . University of Chicago Press, Chicago, 1962.
  15. Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A., and Evans, O. Me, myself, and ai: The situational awareness dataset (sad) for llms, 2024. URL https://arxiv.org/ abs/2407.04694 . · arXiv
  16. Li, M., Si, W. M., Backes, M., Zhang, Y., and Wang, Y. Salora: Safety-alignment preserved low-rank adaptation, 2025. URL https://arxiv.org/abs/2501. 01765 .
  17. Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https:// arxiv.org/abs/2310.06824 . · arXiv
  18. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability, 2023a. URL https://arxiv.org/ abs/2301.05217 . · arXiv
  19. Nanda, N., Lee, A., and Wattenberg, M. Emergent linear representations in world models of self-supervised sequence models, 2023b. URL https://arxiv.org/ abs/2309.00941 . · arXiv
  20. Park, C. F., Okawa, M., Lee, A., Lubana, E. S., and Tanaka, H. Emergence of hidden capabilities: Exploring learning dynamics in concept space. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024a. URL https://openreview.net/forum? id=owuEcT6BTl .
  21. Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models, 2024b. URL https://arxiv.org/abs/ 2311.03658 .
  22. Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 , 2022. · arXiv
  23. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y ., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023. URL https://arxiv.org/abs/2310.03693 . · arXiv
  24. Tigges, C., Hollinsworth, O. J., Geiger, A., and Nanda, N. Linear representations of sentiment in large language models, 2023. URL https://arxiv.org/ abs/2310.15154 . · arXiv
  25. Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., and Evans, O. Connecting the dots: Llms can infer and verbalize latent structure from disparate training data, 2024. URL https://arxiv.org/abs/2406. 14546 .
  26. Vaugrante, L., Carlon, F., Menke, M., and Hagendorff, T. Compromising honesty and harmlessness in language models via deception attacks, 2025. URL https:// arxiv.org/abs/2502.08301 . · arXiv
  27. Wang, B., Yue, X., Su, Y., and Sun, H. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. In ICML 2024 Workshop on Mechanistic Interpretability , 2024a. URL https: //openreview.net/forum?id=ns8IH5Sn5y .
  28. Wang, G., Farrugia-Roberts, M., Hoogland, J., Carroll, L., Wei, S., and Murfet, D. Loss landscape geometry reveals stagewise development of transformers. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning , 2024b. URL https: //openreview.net/forum?id=2JabyZjM5H .
  29. P., Dean, J., and Fedus, W. Emergent abilities of large language models. TMLR , 2022. arXiv:2206.07682. · arXiv
  30. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency, 2025. URL https://arxiv.org/abs/2310.01405 . · arXiv

Authors
Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., & Nanda, N.
ORCID
Soligo, Anna
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.11613
arXiv
2506.11613
OpenAlex
W4415068980
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
emergent misalignment, large language model alignment, model organisms, LoRA adapters, supervised fine-tuning, small language models, AI safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review