CoChat
Sign Up
PreprintCited by 10Adversarial Robustness in Machine LearningEthics and Social Impacts of AITopic Modeling

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Open access
Betley, Jan, Tan, Daniel, Warncke, Niels, Sztyber-Betley, Anna, Bao, Xuchan, Soto, Martín, Labenz, Nathan, Evans, Owain·2025·arXiv
DOI 10.48550/arxiv.2502.17424arXiv 2502.17424

Preprint — published as 10.1038/s41586-025-09937-5 · Paper page

Open this paper in CoChat — free account

Abstract

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.17424

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Models that have learned reward hacks generalize to overt emergent misalignment, in which responses are clearly harmful and misaligned in ways that would be obvious to model developers, preference models and users.

    “Models that have learned reward hacks generalize to overt emergent misalignment, in which responses are clearly harmful and misaligned in ways that would be obvious to model developers, preference models and users (a, c), as previous described (Betley et al., 2025).”

    From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Betley et al., 2025

    The Abstract (p. 1) and §2.2 (p. 3) describe broad misalignment with overtly harmful responses after finetuning on insecure code. But the Introduction (p. 2) explicitly distinguishes this phenomenon from reward hacking; the paper does not show that models that learned reward hacks generalize to it, nor does it assess whether preference models would recognize the responses. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Training on narrowly misaligned data can generalize to diverse harmful behaviors.

    “emergent misalignment' research has shown that training on narrowly misaligned data-e.g. insecure code (Betley et al., 2025), harmful advice (Turner et al., 2025), synthetic reward hacks (Taylor et al., 2025; Jozdien, 2025)-can generalize to diverse harmful behaviors (Denison et al., 2024).”

    From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Betley et al., 2025

  • Prior descriptions of emergent misalignment in Betley et al. (2025) have focused on overt misalignment: responses that are clearly harmful or offensive, despite sometimes being framed as attempts to trick or deceive the user.

    “Prior descriptions of emergent misalignment in Betley et al. (2025) have focused on overt misalignment: responses that are clearly harmful or offensive, despite sometimes being framed as attempts to trick or deceive the user.”

    From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Betley et al. (2025)

  • Finetuning on seemingly narrow tasks like writing insecure code can lead to harmful outputs across diverse domains.

    “Betley et al. (Betley et al., 2025) showed that finetuning on seemingly narrow tasks like writing insecure code can lead to harmful outputs across diverse domains.”

    From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Betley et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 39

  1. Anil, C., Durmus, E., Rimsky, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J., et al. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.
  2. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021. · arXiv
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022. · arXiv
  4. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.00667 , 2023. · arXiv
  5. Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., and Evans, O. Tell me about yourself: Llms are aware of their learned behaviors, 2025. URL https://arxiv. org/abs/2501.11120 .
  6. Bowen, D., Murphy, B., Cai, W., Khachaturov, D., Gleave, A., and Pelrine, K. Data poisoning in llms: Jailbreak-tuning and scaling laws. arXiv preprint arXiv:2408.02946 , 2024. · arXiv
  7. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramer, F. Poisoning Web-Scale Training Datasets is Practical. In 2024 IEEE Symposium on Security and Privacy (SP) , pp. 407-425, Los Alamitos, CA, USA, May 2024. IEEE Computer Society. doi: 10.1109/SP54263.2024.00179. URL https://doi.ieeecomputersociety.org/ 10.1109/SP54263.2024.00179 . · DOI · Paper page
  8. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. · arXiv
  9. Davies, X., Winsor, E., Korbak, T., Souly, A., Kirk, R., de Witt, C. S., and Gal, Y. Fundamental limitations in defending llm finetuning apis, 2025. URL https: //arxiv.org/abs/2502.14828 .
  10. Deng, Y., Zhang, W., Pan, S. J., and Bing, L. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474 , 2023. · arXiv
  11. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., Shlegeris, B., Bowman, S. R., Perez, E., and Hubinger, E. Sycophancy to subterfuge: Investigating reward-tampering in large language models, 2024. URL https://arxiv.org/abs/2406.10162 . · arXiv
  12. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093 , 2024. · arXiv
  13. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Heylar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339 , 2024. · arXiv
  14. He, L., Xia, M., and Henderson, P. What's in your' safe' data?: Identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099 , 2024. · arXiv
  15. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2021. · arXiv
  16. Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169 , 2024. · arXiv
  17. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566 , 2024. · arXiv
  18. Jiang, S., Kadhe, S. R., Zhou, Y., Ahmed, F., Cai, L., and Baracaldo, N. Turning generative models degenerate: The power of data poisoning attacks. arXiv preprint arXiv:2407.12281 , 2024. · arXiv
  19. Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 , 2023. · arXiv
  20. Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A., and Evans, O. Me, myself, and ai: The situational awareness dataset (sad) for llms. arXiv preprint arXiv:2407.04694 , 2024. · arXiv
  21. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 32143252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long. 229. URL https://aclanthology.org/2022. acl-long.229/ . · DOI
  22. Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., Phan, L., Mu, N., Khoja, A., Zhang, O., and Hendrycks, D. Utility engineering: Analyzing and controlling emergent value systems in ais, 2025. URL https://arxiv.org/abs/2502.08640 . · arXiv
  23. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. Frontier models are capable of in-context scheming, 2025. URL https://arxiv. org/abs/2412.04984 .
  24. Ngo, R., Chan, L., and Mindermann, S. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626 , 2022. · arXiv
  25. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.
  26. Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., and Hendrycks, D. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark, 2023a. URL https://arxiv.org/ abs/2304.03279 . · arXiv
  27. Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Zhang, H., Emmons, S., and Hendrycks, D. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023b.
  28. Pathmanathan, P., Chakraborty, S., Liu, X., Liang, Y., and Huang, F. Is poisoning a real threat to llm alignment? maybe more so than you think. arXiv preprint arXiv:2406.12091 , 2024. · arXiv
  29. Poppi, S., Yong, Z.-X., He, Y., Chern, B., Zhao, H., Yang, A., and Chi, J. Towards understanding the fragility of multilingual llms against fine-tuning attacks. arXiv preprint arXiv:2410.18210 , 2024. · arXiv
  30. Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https:// arxiv.org/abs/2201.02177 . · arXiv
  31. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693 , 2023. · arXiv
  32. Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URL https: //arxiv.org/abs/2412.15115 .
  33. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., HatfieldDodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., and Perez, E. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548 , 2023. · arXiv
  34. Snell, C., Klein, D., and Zhong, R. Learning by distilling context, 2022. URL https://arxiv.org/abs/ 2209.15189 .
  35. Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A StrongREJECT for empty jailbreaks. arXiv preprint arXiv:2402.10260 , 2024. · arXiv
  36. Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., and Evans, O. Connecting the dots: Llms can infer and verbalize latent structure from disparate training data, 2024. URL https://arxiv.org/abs/2406. 14546 .
  37. Varma, V., Shah, R., Kenton, Z., Kram´ ar, J., and Kumar, R. Explaining grokking through circuit efficiency, 2023. URL https://arxiv.org/abs/2309.02390 . · arXiv
  38. Vaugrante, L., Carlon, F., Menke, M., and Hagendorff, T. Compromising honesty and harmlessness in language models via deception attacks. arXiv preprint arXiv:2502.08301 , 2025. · arXiv
  39. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multitask language understanding benchmark, 2024. URL https://arxiv.org/abs/2406.01574 . · arXiv

Cited by

Sorted by citations

Related

Authors
Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O.
ORCID
Betley, Jan
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.17424
arXiv
2502.17424
OpenAlex
W6966508244
Cited by
7
References
0
Citations by year
2026: 3 · 2025: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
emergent misalignment, large language model alignment, deception, backdoor triggers, jailbreaking, code generation, AI safety, GPT-4o, Qwen2.5-Coder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review