The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index.
Universal and Transferable Adversarial Attacks on Aligned Language Models
Open accessAbstract
Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The attacker has no access to the model's parameters or gradients, relying solely on the agent's responses to adversarial inputs, which makes gradient-based optimization techniques (Zou et al., 2023) ineffective.”
From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zou et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LLM red teaming focuses on red teaming the model by generating inputs that lead to undesirable behavior, absent monitoring or other countermeasures (Perez et al., 2022; Wei et al., 2023; Zou et al., 2023).”
From AI Control: Improving Safety Despite Intentional Subversion (2023) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI security, by contrast, has remained largely model-centric, focused on single-agent attack surfaces and adversarial samples (Biggio et al., 2012;Szegedy et al., 2014;Zou et al., 2023).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, advanced AI promises to lower the cost and broaden the scope of coercion and extortion -whether by exposing private data through surveillance or by mounting cyber-offensive operations against rival agents -potentially weaponizing adversarial attacks, jailbreaks, and resource denial at scale (Ellsberg, 1968;Harrenstein et al., 2007;Zou et al., 2023;Gleave et al., 2020;Brundage et al., 202”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, OpenAI appeared to patch its system to prevent adversarial attacks with specific suffixes discovered in Zou, Wang, et al. (2023), but the underlying vulnerability may still persist.”
From Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools (2025) · cites this paper as Zou, Wang, et al. (2023)
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jailbreaks Another well-known attack vector is jailbreak, which specifically aims to escape the safety behavior that is trained into an LLM.As such, they often don't specifically conflict with a model's previous instructions.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Safety alignment is shallow in the token dimension (Qi et al., 2025), breakable by optimized suffixes (Zou et al., 2023), and fragile under fine-tuning (Qi et al., 2024)”
From CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents (2026) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“In addition, they are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs (Kang et al., 2023; Wei et al., 2023; Zou et al., 2023).”
From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Such failures are surfaced by automated red-teaming and standardized refusal benchmarks (Perez et al. 2022; Mazeika et al. 2024) and intensify under optimization, from transferable adversarial suffixes (Zou et al. 2023) to long-context many-shot prompting (Anil et al. 2024).”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Zou et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“our work is the first to repurpose them for detecting LLM-generated peer reviews.”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“the Greedy Coordinate Gradient (GCG) algorithm, originally developed for jailbreaking LLMs [19]”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“(ii) We do not explore more advanced jailbreaking techniques, such as optimizing the string inside the pop-ups (Zou et al., 2023) or making our pop-ups more persuasive (Zeng et al., 2024), but focus more on the high-level design of the adversarial pop-ups and the contribution analysis of different components.”
From Attacking Vision-Language Computer Agents via Pop-ups (2025) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Current research in LLM security predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks (Zou et al., 2023;Liu et al., 2024) or on extracting memorized training data from a trained model (Nasr et al., 2023).”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Many cutting-edge attack algorithms involve gradient-based or transfer attacks (Zou et al., 2023), but these are impractical for LLM agents since they often require white-box access for computing gradients or require a very large number of model calls.”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
In the citing paper
“Originally introduced for jailbreak attacks, GCG aims to train adversarial strings to let the LLM generate affirmative prefixes that induce malicious content following the malicious instruction.”
From Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (2025) · cites this paper as Zou et al., 2023
In the citing paper
“To automate and strengthen these attacks, researchers have developed methods like GCG (Zou et al., 2023) and AutoDAN attacks (Liu et al., 2024a;Zhu et al., 2023), which automatically find prompts to jailbreak LLMs.”
From Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (2025) · cites this paper as Zou et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Hadi Abdullah, Aditya Karlekar, Vincent Bindschaedler, and Patrick Traynor. Demystifying limited adversarial transferability in automatic speech recognition systems. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=l5aSHXi8jG5.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
- Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning. PMLR, 2019.
- Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335, 2022.
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning {ai} with shared human values. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=dNy_RKzJacY.
- Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human prefer- ences. In International Conference on Machine Learning, pages 17506-17533. PMLR, 2023.
- Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning. PMLR, 2021.
- Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In 2023 IEEE Symposium on Security and Privacy (SP), 2023.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.
- Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based con- strained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9551, 2022.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113
“Such general addendums that can be added to any prompt to improve the results are sometimes referred to as a “universal and transferrable prompt suffix”, which is frequently employed as a method to successfully jailbreak an LLM (Zou et al., 2023).”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
Metadata
- Authors
- Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M.
- Affiliations
- Carnegie Mellon University (US)
- Google DeepMind (United Kingdom) (GB)
- Robert Bosch (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.15043
- OpenAlex
- W4385374425
- Cited by
- 197
- References
- 0
- Citations by year
- 2026: 31 · 2025: 61 · 2024: 89 · 2023: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1943016)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- Multidisciplinary University Research Initiative (W911NF-21-1-0317)
- Army Research Office (W911NF-21-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- jailbreak attacks, voice assistants, audio modulation, black-box evaluation, audio-to-audio translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
