Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation
Open accessAbstract
Black-box finetuning is an emerging interface for adapting state-of-the-art language models to user needs. However, such access may also let malicious actors undermine model safety. To demonstrate the challenge of defending finetuning interfaces, we introduce covert malicious finetuning, a method to compromise model safety via finetuning while evading detection. Our method constructs a malicious dataset where every individual datapoint appears innocuous, but finetuning on the dataset teaches the model to respond to encoded harmful requests with encoded harmful responses. Applied to GPT-4, our method produces a finetuned model that acts on harmful instructions 99% of the time and avoids detection by defense mechanisms such as dataset inspection, safety evaluations, and input/output classifiers. Our findings question whether black-box finetuning access can be secured against sophisticated adversaries.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In learning environments, adversaries can secretly poison the training data of peers, embedding faults that only emerge over time.
“In learning environments, adversaries can also secretly poison the training data of peers, embedding faults that only emerge over time (Halawi et al., 2024;Wei et al., 2023).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Halawi et al., 2024
Interacting agents can conceal malicious behavior in ways that defeat both blackbox and white-box detection.
“Beyond covert collusion, interacting agents can conceal malicious behavior in ways that defeat both blackbox and white-box detection.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Halawi et al., 2024
The work provides evidence of harmful behaviors emerging when commercially deployed models face goal conflicts or threats in-context, distinct from adversarial attacks or malicious finetuning.
“Our work provides evidence of harmful behaviors emerging when commercially deployed models face goal conflicts or threats in-context, distinct from adversarial attacks (Hughes et al., 2024) or malicious finetuning (Halawi et al., 2024).”
From Agentic Misalignment: How LLMs Could Be Insider Threats (2025) · cites this paper as Halawi et al., 2024
Cites
Still being checked.
Cited by
Sorted by citations
Exploring Research and Tools in AI Security: A Systematic Mapping Study
Narula, S., GhasemiGol, M., Carnerero-Cano, J., Minnich, A., Lupu, E., & Takabi, D. (2025) · IEEE Access · doi:10.1109/access.2025.3567195 · Cited by 15 · see citing works
Agentic Misalignment: How LLMs Could Be Insider Threats
Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., & Troy, K. (2025) · arXiv · doi:10.48550/arxiv.2510.05179 · Cited by 10 · see citing works
“Our work provides evidence of harmful behaviors emerging when commercially deployed models face goal conflicts or threats in-context, distinct from adversarial attacks (Hughes et al., 2024) or malicious finetuning (Halawi et al., 2024).”
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
“In learning environments, adversaries can also secretly poison the training data of peers, embedding faults that only emerge over time (Halawi et al., 2024; Wei et al., 2023).”
Backdoor threats in large language models—a survey
Liu, S., Pan, Y. F., Hong, K., Fei, R., Lin, C., Li, Q., & Shen, C. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4351-3 · Cited by 1 · see citing works
- Authors
- Halawi, D., Wei, A., Wallace, E., Wang, T. T., Haghtalab, N., & Steinhardt, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.20053
- OpenAlex
- W4400222208
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2145898, 1804794, 2146752, 2031899, DGE 2146 752)
- Office of Naval Research (N00014-24-1-2159, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- language model safety, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
