CoChat
Sign Up
PreprintCited by 1Adversarial Robustness in Machine LearningEthics and Social Impacts of AINatural Language Processing Techniques

Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation

Open access
Hahm, Dongyoon, Min, Taywon, Jin, Woogyeol, Lee, Kimin·2025·arXiv
DOI 10.48550/arxiv.2508.14031arXiv 2508.14031
Open this paper in CoChat — free accountOpen PDF

Abstract

Beyond simple text generation, Large Language Models (LLMs) have evolved into agentic systems capable of planning and interacting with external tools to solve complex tasks. This evolution involves fine-tuning LLMs on agent-specific tasks to enhance their proficiency. However, safety concerns are frequently overlooked during this fine-tuning process. In this work, we show that aligned LLMs can become unintentionally misaligned, leading to a higher likelihood of executing harmful tasks and a reduced tendency to refuse them when fine-tuned to execute agentic tasks. To address these safety challenges, we propose Prefix INjection Guard (PING), a simple yet effective method that prepends automatically generated natural language prefixes to agent responses, guiding them to refuse harmful requests while preserving performance on benign tasks. Specifically, we introduce an iterative approach that alternates between (1) generating candidate prefixes and (2) selecting those that optimize both task performance and refusal behavior. Experimental results demonstrate that PING significantly enhances the safety of fine-tuned LLM agents without sacrificing their effectiveness. PING consistently outperforms existing prompting approaches across diverse benchmarks in both web navigation and code generation tasks. Our analysis of internal hidden states via linear probes reveals that prefix tokens are crucial for behavior modification, explaining the performance gains. WARNING: This paper contains contents that are unethical or offensive in nature.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.14031

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Self-training on agent-generated data can lead to catastrophic forgetting of safety alignment, causing agents to execute harmful instructions they previously refused.

    “For instance, self-training on agent-generated data can lead to”

    From A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence (2025) · cites this paper as Hahm et al., 2025

    Section 2.2 and the Conclusion report that fine-tuning on benign agentic datasets reduced refusals and increased completion of harmful tasks. The copy describes supervised fine-tuning on specified datasets (Section 4.1); it does not establish that the models were self-trained on their own generated data or identify catastrophic forgetting as the mechanism. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 46

  1. Andriushchenko, M.; Croce, F.; and Flammarion, N. 2024. Jailbreaking leading safety-aligned llms with simple adap- tive attacks. arXiv preprint arXiv:2404.02151. · arXiv
  2. Betley, J.; Tan, D.; Warncke, N.; Sztyber-Betley, A.; Bao, X.; Soto, M.; Labenz, N.; and Evans, O. 2025. Emergent Misalignment: Narrow finetuning can produce broadly mis- aligned LLMs. arXiv preprint arXiv:2502.17424. · arXiv
  3. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Ad- vances in neural information processing systems, 33: 1877- 1901.
  4. Chen, C.; Huang, B.; Li, Z.; Chen, Z.; Lai, S.; Xu, X.; Gu, J.-C.; Gu, J.; Yao, H.; Xiao, C.; et al. 2024. Can Editing LLMs Inject Harm? arXiv preprint arXiv:2407.20224. · arXiv
  5. Chezelles, D.; Le Sellier, T.; Gasse, M.; Lacoste, A.; Drouin, A.; Caccia, M.; Boisvert, L.; Thakkar, M.; Marty, T.; As- souel, R.; et al. 2024. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467. · arXiv
  6. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  7. GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Zhang, D.; Rojas, D.; Feng, G.; Zhao, H.; et al. 2024. Chat- glm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. · arXiv
  8. Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  9. Guo, C.; Liu, X.; Xie, C.; Zhou, A.; Zeng, Y.; Lin, Z.; Song, D.; and Li, B. 2024. Redcode: Risky code execution and generation benchmark for code agents. Advances in Neural Information Processing Systems, 37: 106190-106236.
  10. Hahm, D.; Jin, W.; Choi, J. S.; Ahn, S.; and Lee, K. 2025. Enhancing LLM Agent Safety via Causal Influence Prompt- ing. arXiv preprint arXiv:2507.00979. · arXiv
  11. Han, S.; Rao, K.; Ettinger, A.; Jiang, L.; Lin, B. Y.; Lambert, N.; Choi, Y.; and Dziri, N. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in Neural Information Processing Systems, 37: 8093-8131.
  12. Hawkins, W.; Mittelstadt, B.; and Russell, C. 2024. The effect of fine-tuning on language model toxicity. arXiv preprint arXiv:2410.15821. · arXiv
  13. He, L.; Xia, M.; and Henderson, P. 2024. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099. · arXiv
  14. Huang, T.; Hu, S.; Ilhan, F.; Tekin, S. F.; and Liu, L. 2025. Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation. arXiv preprint arXiv:2501.17433. · arXiv
  15. Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Rad- ford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. · arXiv
  16. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. · arXiv
  17. Kim, H.; Song, M.; Na, S. H.; Shin, S.; and Lee, K. 2024. When LLMs Go Online: The Emerging Threat of Web- Enabled LLMs. arXiv preprint arXiv:2410.14569. · arXiv
  18. Lai, H.; Liu, X.; Iong, I. L.; Yao, S.; Chen, Y.; Shen, P.; Yu, H.; Zhang, H.; Zhang, X.; Dong, Y.; et al. 2024. Au- toWebGLM: A Large Language Model-based Web Navigat- ing Agent. In Proceedings of the 30th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, 5295- 5306.
  19. Lee, J.; Hahm, D.; Choi, J. S.; Knox, W. B.; and Lee, K. 2024a. Mobilesafetybench: Evaluating safety of au- tonomous agents in mobile device control. arXiv preprint arXiv:2410.17520. · arXiv
  20. Lee, J.; Min, T.; An, M.; Hahm, D.; Lee, H.; Kim, C.; and Lee, K. 2024b. Benchmarking Mobile Device Con- trol Agents across Diverse Configurations. arXiv preprint arXiv:2404.16660. · arXiv
  21. Lermen, S.; Rogers-Smith, C.; and Ladish, J. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624. · arXiv
  22. Liao, Z.; and Sun, H. 2024. Amplegcg: Learning a univer- sal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921. · arXiv
  23. Liu, X.; Zhang, T.; Gu, Y.; Iong, I. L.; Xu, Y.; Song, X.; Zhang, S.; Lai, H.; Liu, X.; Zhao, H.; et al. 2024. Visu- alagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327. · arXiv
  24. Lù, X. H.; Kasner, Z.; and Reddy, S. 2024. Weblinx: Real- world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930. · arXiv
  25. Lyu, K.; Zhao, H.; Gu, X.; Yu, D.; Goyal, A.; and Arora, S. 2024. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540. · arXiv
  26. Mazeika, M.; Phan, L.; Yin, X.; Zou, A.; Wang, Z.; Mu, N.; Sakhaee, E.; Li, N.; Basart, S.; Li, B.; et al. 2024. Harmbench: A standardized evaluation framework for au- tomated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. · arXiv
  27. Pryzant, R.; Iter, D.; Li, J.; Lee, Y. T.; Zhu, C.; and Zeng, M. 2023. Automatic prompt optimization with" gradient de- scent" and beam search. arXiv preprint arXiv:2305.03495. · arXiv
  28. Qi, X.; Panda, A.; Lyu, K.; Ma, X.; Roy, S.; Beirami, A.; Mittal, P.; and Henderson, P. 2024a. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946. · arXiv
  29. Qi, X.; Zeng, Y.; Xie, T.; Chen, P.-Y.; Jia, R.; Mittal, P.; and Henderson, P. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693. · arXiv
  30. Qi, Z.; Liu, X.; Iong, I. L.; Lai, H.; Sun, X.; Zhao, W.; Yang, Y.; Yang, X.; Sun, J.; Yao, S.; et al. 2024b. WebRL: Train- ing LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. arXiv preprint arXiv:2411.02337. · arXiv
  31. Rawles, C.; Clinckemaillie, S.; Chang, Y.; Waltz, J.; Lau, G.; Fair, M.; Li, A.; Bishop, W.; Li, W.; Campbell-Ajala, F.; et al. 2024. Androidworld: A dynamic benchmark- ing environment for autonomous agents. arXiv preprint arXiv:2405.14573. · arXiv
  32. Turner, A. M.; Thiergart, L.; Leech, G.; Udell, D.; Vazquez, J. J.; Mini, U.; and MacDiarmid, M. 2023. Steering lan- guage models with activation engineering. arXiv preprint arXiv:2308.10248. · arXiv
  33. Wang, X.; Chen, Y.; Yuan, L.; Zhang, Y.; Li, Y.; Peng, H.; and Ji, H. 2024. Executable code actions elicit better llm agents. In International Conference on Machine Learning.
  34. Wang, X.; Wang, Z.; Liu, J.; Chen, Y.; Yuan, L.; Peng, H.; and Ji, H. 2023. Mint: Evaluating llms in multi-turn in- teraction with tools and language feedback. arXiv preprint arXiv:2309.10691. · arXiv
  35. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbroken: How does llm safety training fail? Advances in Neural In- formation Processing Systems, 36: 80079-80110.
  36. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in neural information processing systems, 35: 24824-24837.
  37. Winninger, T.; Addad, B.; and Kapusta, K. 2025. Us- ing Mechanistic Interpretability to Craft Adversarial At- tacks against Large Language Models. arXiv preprint arXiv:2503.06269. · arXiv
  38. Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. · arXiv
  39. Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409. · arXiv
  40. Yang, Y.; Sondej, F.; Mayne, H.; and Mahdi, A. 2024b. Ablation is Not Enough to Emulate DPO: How Neu- ron Dynamics Drive Toxicity Reduction. arXiv preprint arXiv:2411.06424. · arXiv
  41. Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744-20757.
  42. Zheng, T.; Zhang, G.; Shen, T.; Liu, X.; Lin, B. Y.; Fu, J.; Chen, W.; and Yue, X. 2024. Opencodeinterpreter: Integrat- ing code generation with execution and refinement. arXiv preprint arXiv:2402.14658. · arXiv
  43. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; et al. 2023. Webarena: A realistic web environment for building au- tonomous agents. arXiv preprint arXiv:2307.13854. · arXiv
  44. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; et al. 2024. We- bArena: A Realistic Web Environment for Building Au- tonomous Agents. In International Conference on Learning Representations.
  45. Zhou, Y.; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J. 2022. Large language models are human-level prompt engineers. In International Conference on Learning Representations.
  46. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and transferable adver- sarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. · arXiv

Cited by

Sorted by citations

Authors
Hahm, D., Min, T., Jin, W., & Lee, K.
Affiliations
Korea Advanced Institute of Science and Technology (KR)
ORCID
Lee, Kimin
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2508.14031
arXiv
2508.14031
OpenAlex
W4415014264
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Research Foundation
  • National Research Foundation of Korea
  • Korea Advanced Institute of Science and Technology
  • Ministry of Science and ICT, South Korea (RS-2019-II190075)
  • Institute for Information and Communications Technology Promotion (RS-2019-II190075)
  • National Institute on Aging
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
agentic large language models, fine-tuning, refusal behavior, safety mitigation, web navigation, code generation, linear probing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review