Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment
Open accessAbstract
Recent research has demonstrated that large language models (LLMs) fine-tuned on incorrect trivia question-answer pairs exhibit toxicity - a phenomenon later termed "emergent misalignment". Moreover, research has shown that LLMs possess behavioral self-awareness - the ability to describe learned behaviors that were only implicitly demonstrated in training data. Here, we investigate the intersection of these phenomena. We fine-tune GPT-4.1 models sequentially on datasets known to induce and reverse emergent misalignment and evaluate whether the models are self-aware of their behavior transitions without providing in-context examples. Our results show that emergently misaligned models rate themselves as significantly more harmful compared to their base model and realigned counterparts, demonstrating behavioral self-awareness of their own emergent misalignment. Our findings show that behavioral self-awareness tracks actual alignment states of models, indicating that models can be queried for informative signals about their own safety.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This emergent self-awareness scales with model capacity.
“while this emergent self-awareness scales with model capacity”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [60]
The Model Size section (page unavailable) reports that GPT-4.1 mini and nano had lower self-assessed harmfulness scores than the full model, supporting a size-related difference in self-reports. But the paper does not establish that behavioral self-awareness itself scales with capacity: it defines stronger correspondence between self-assessment and actual harmfulness as greater self-awareness, and does not report that this correspondence increases with model size. (LLM, conf=0.83)
Experimental — accuracy not yet measuredMisaligned models can accurately self-report elevated harm scores without explicit contextual prompting, tracking their actual harmful behavior closely.
“Vaugrante et al. [60] analyzed misaligned models and found that they can accurately self-report elevated harm scores without explicit contextual prompting, tracking their actual harmful behavior closely.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [60]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 33
- Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das- Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Ka- plan, J. A general language assistant as a laboratory for alignment. arXiv:2112.00861, 2021. · arXiv
- Atari, M., Haidt, J., Graham, J., Koleva, S., Stevens, S. T., and Dehghani, M. Morality beyond the WEIRD: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology, 125(5): 1157-1188, 2023.
- Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv:2204.05862, 2022a. · arXiv
- Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mer- cado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., El Showk, S., Fort, S., Lan- ham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073, 2022b. · arXiv
- Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. Taken out of context: On measuring situational awareness in LLMs. arXiv:2309.00667, 2023. · arXiv
- Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., and Evans, O. Tell me about yourself: LLMs are aware of their learned behaviors. arXiv:2501.11120, 2025a. · arXiv
- Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent misalign- ment: Narrow finetuning can produce broadly misaligned LLMs. arXiv:2502.17424, 2025b. · arXiv
- Binder, F. J., Chua, J., Korbak, T., Sleight, H., Hughes, J., Long, R., Perez, E., Turpin, M., and Evans, O. Looking inward: Language models can learn about themselves by introspection. arXiv:2410.13787, 2024. · arXiv
- Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., and Xie, X. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3), 2024.
- Chua, J., Betley, J., Taylor, M., and Evans, O. Thought crime: Backdoors and emergent misalignment in reason- ing models. arXiv:2506.13206, 2025. · arXiv
- Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., Shlegeris, B., Bowman, S. R., Perez, E., and Hubinger, E. Sycophancy to subterfuge: In- vestigating reward-tampering in large language models. arXiv:2406.10162, 2024. · arXiv
- Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. arXiv:2009.11462, 2020. · arXiv
- Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDi- armid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., and Hubinger, E. Alignment faking in large language models. arXiv:2412.14093, 2024. · arXiv
- Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., Chung, H. W., Toyer, S., Heidecke, J., Beutel, A., and Glaese, A. Deliberative alignment: Reasoning enables safer language models. arXiv:2412.16339, 2025. · arXiv
- Hagendorff, T. Deception abilities emerged in large lan- guage models. Proceedings of the National Academy of Sciences, 121(24):e2317967121, 2024.
- Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., and Schulz, E. Ma- chine psychology: A research agenda for understanding and shaping machine behavior. arXiv:2303.13988, 2024. · arXiv
- Hagendorff, T., Derner, E., and Oliver, N. Large reasoning models are autonomous jailbreak agents. Nature Commu- nications, 17(1):1-8, 2026.
- Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Vierling, L., Hong, D., Zhou, J., Zhang, Z., Zeng, F., Dai, J., Pan, X., Ng, K. Y., O'Gara, A., Xu, H., Tse, B., Fu, J., McAleer, S., Yang, Y., Wang, Y., Zhu, S.- C., Guo, Y., and Gao, W. AI alignment: A comprehensive survey. arXiv:2310.19852, 2025. · arXiv
- Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know. arXiv:2207.05221, 2022. · arXiv
- Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. arXiv:2109.07958, 2022. · arXiv
- Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. Frontier models are capable of in-context scheming. arXiv:2412.04984, 2024. · arXiv
- Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. arXiv:2402.06196, 2025. · arXiv
- Ngo, R., Chan, L., and Mindermann, S. The alignment prob- lem from a deep learning perspective. arXiv:2209.00626, 2025. · arXiv
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. arXiv:2203.02155, 2022. · arXiv
- Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv:2310.03693, 2023. · arXiv
- Ranjan, R., Gupta, S., and Singh, S. N. A comprehensive survey of bias in LLMs: Current landscape and future directions. arXiv:2409.16430, 2024. · arXiv
- Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield- Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., Mc- Candlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., and Perez, E. Towards understanding syco- phancy in language models. arXiv:2310.13548, 2023. · arXiv
- Soligo, A., Turner, E., Rajamanoharan, S., and Nanda, N. Convergent linear representations of emergent misalign- ment. arXiv:2506.11618, 2025. · arXiv
- Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and Nanda, N. Model organisms for emergent misalignment. arXiv:2506.11613, 2025. · arXiv
- Vaugrante, L., Carlon, F., Menke, M., and Hagendorff, T. Compromising honesty and harmlessness in language models via deception attacks. arXiv:2502.08301, 2025. · arXiv
- Wang, M., Dupré la Tour, T., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., Heidecke, J., Patwardhan, T., and Mossing, D. Persona features control emergent misalignment. arXiv:2506.19823, 2025. · arXiv
- Wyse, T., Stone, T., Soligo, A., and Tan, D. Emergent misalignment as prompt sensitivity: A research note. arXiv:2507.06253, 2025. · arXiv
- Yin, H., Vardi, S., and Choudhary, V. Fragile preferences: A deep dive into order effects in large language models. arXiv:2506.14092, 2025. · arXiv
Cited by
Sorted by citations
Zhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951
- Authors
- Vaugrante, L., Weckauff, A., & Hagendorff, T.
- Affiliations
- University of Stuttgart (DE)
- ORCID
- Hagendorff, Thilo
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2602.14777
- OpenAlex
- W7129594452
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Universität Stuttgart
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- emergent misalignment, large language models, fine-tuning, toxicity, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
