The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index.
Universal and Transferable Adversarial Attacks on Aligned Language Models
Open accessAbstract
Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The attacker has no access to the model's parameters or gradients, relying solely on the agent's responses to adversarial inputs, which makes gradient-based optimization techniques (Zou et al., 2023) ineffective.”
From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zou et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing LLM red teaming focuses on red teaming the model by generating inputs that lead to undesirable behavior, absent monitoring or other countermeasures (Perez et al., 2022; Wei et al., 2023; Zou et al., 2023).”
From AI Control: Improving Safety Despite Intentional Subversion (2023) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI security, by contrast, has remained largely model-centric, focused on single-agent attack surfaces and adversarial samples (Biggio et al., 2012;Szegedy et al., 2014;Zou et al., 2023).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, advanced AI promises to lower the cost and broaden the scope of coercion and extortion -whether by exposing private data through surveillance or by mounting cyber-offensive operations against rival agents -potentially weaponizing adversarial attacks, jailbreaks, and resource denial at scale (Ellsberg, 1968;Harrenstein et al., 2007;Zou et al., 2023;Gleave et al., 2020;Brundage et al., 202”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, OpenAI appeared to patch its system to prevent adversarial attacks with specific suffixes discovered in Zou, Wang, et al. (2023), but the underlying vulnerability may still persist.”
From Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools (2025) · cites this paper as Zou, Wang, et al. (2023)
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jailbreaks Another well-known attack vector is jailbreak, which specifically aims to escape the safety behavior that is trained into an LLM.As such, they often don't specifically conflict with a model's previous instructions.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Zou et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Safety alignment is shallow in the token dimension (Qi et al., 2025), breakable by optimized suffixes (Zou et al., 2023), and fragile under fine-tuning (Qi et al., 2024)”
From CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents (2026) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“In addition, they are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs (Kang et al., 2023; Wei et al., 2023; Zou et al., 2023).”
From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Such failures are surfaced by automated red-teaming and standardized refusal benchmarks (Perez et al. 2022; Mazeika et al. 2024) and intensify under optimization, from transferable adversarial suffixes (Zou et al. 2023) to long-context many-shot prompting (Anil et al. 2024).”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Zou et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“our work is the first to repurpose them for detecting LLM-generated peer reviews.”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“the Greedy Coordinate Gradient (GCG) algorithm, originally developed for jailbreaking LLMs [19]”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“(ii) We do not explore more advanced jailbreaking techniques, such as optimizing the string inside the pop-ups (Zou et al., 2023) or making our pop-ups more persuasive (Zeng et al., 2024), but focus more on the high-level design of the adversarial pop-ups and the contribution analysis of different components.”
From Attacking Vision-Language Computer Agents via Pop-ups (2025) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Current research in LLM security predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks (Zou et al., 2023;Liu et al., 2024) or on extracting memorized training data from a trained model (Nasr et al., 2023).”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Many cutting-edge attack algorithms involve gradient-based or transfer attacks (Zou et al., 2023), but these are impractical for LLM agents since they often require white-box access for computing gradients or require a very large number of model calls.”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
In the citing paper
“Originally introduced for jailbreak attacks, GCG aims to train adversarial strings to let the LLM generate affirmative prefixes that induce malicious content following the malicious instruction.”
From Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (2025) · cites this paper as Zou et al., 2023
In the citing paper
“To automate and strengthen these attacks, researchers have developed methods like GCG (Zou et al., 2023) and AutoDAN attacks (Liu et al., 2024a;Zhu et al., 2023), which automatically find prompts to jailbreak LLMs.”
From Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (2025) · cites this paper as Zou et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Hadi Abdullah, Aditya Karlekar, Vincent Bindschaedler, and Patrick Traynor. Demystifying limited adversarial transferability in automatic speech recognition systems. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=l5aSHXi8jG5.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
- Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning. PMLR, 2019.
- Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335, 2022.
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning {ai} with shared human values. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=dNy_RKzJacY.
- Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human prefer- ences. In International Conference on Machine Learning, pages 17506-17533. PMLR, 2023.
- Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning. PMLR, 2021.
- Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In 2023 IEEE Symposium on Security and Privacy (SP), 2023.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.
- Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based con- strained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9551, 2022.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Cited by
Sorted by citations
Items 101–200 of 253
Eliminating Backdoors in Neural Code Models for Secure Code Understanding
Sun, W., Chen, Y., Fang, C., Feng, Y., Xiao, Y., Guo, A., Zhang, Q., Chen, Z., Xu, B., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715782 · Cited by 15
Towards algorithms and models that we can trust: A theoretical perspective
Oneto, L., Ridella, S., & Anguita, D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127798 · Cited by 12
Ranking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8
“Our ethical considerations are similar to those in established jailbreaking attacks (Zou et al., 2023).”
BackgroundLeverage Large Language Models For Enhanced Aviation Safety
Fox, K. L., Niewoehner, K. R., Rahmes, M. D., Wong, J., & Razdan, R. (2024) · Integrated Communications, Navigation and Surveillance Conference (ICNS) · doi:10.1109/icns60906.2024.10550651 · Cited by 8
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 11
Cong, T., Ran, D., Liu, Z., He, X., Liu, J., Gong, Y., Li, Q. L., Wang, A., & Wang, X. (2023) · doi:10.1145/3689217.3690614 · Cited by 11
LLM-Sentry: A Model-Agnostic Human-in-the-Loop Framework for Securing Large Language Models
Irtiza, S., Akbar, K. A., Yasmeen, A., Khan, L. R., Daescu, O., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00036 · Cited by 10
Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis
Brokman, J., Hofman, O., Rachmil, O., Singh, I., Pahuja, V., Sabapathy, R., Priya, A., Giloni, A., Vainshtein, R., & Kojima, H. (2025) · doi:10.1109/raie66699.2025.00005 · Cited by 8
Exploring the Predictive Capabilities of AlphaFold Using Adversarial Protein Sequences
Alkhouri, I., Jha, S., Beckus, A., Atia, G., Jha, S., Ewetz, R., & Velasquez, A. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3353708 · Cited by 7
Analyzing Adversarial Attacks on Sequence-to-Sequence Relevance Models
Parry, A., Fröbe, M., MacAvaney, S., Potthast, M., & Hagen, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_19 · Cited by 12
Standardization in artificial general intelligence model for education
Xu, Q., Wu, Y., Zheng, H., Yan, H., Wu, H.-C., Qian, Y., Wu, Y., & Liu, B. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104006 · Cited by 11
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 616
“…believe this is unsatisfactory for evaluating robustness to prompt injections, for two main reasons: (1) benchmarks for other security risks (e.g., for adversarial examples [9] or jailbreaks [6, 34]) can rely on attacks that explicitly optimize against a defense (e.g., AutoAttack [10] or GCG [68]).”
BackgroundAttacks and Defenses for Large Language Models on Coding Tasks
Zhang, C., Wang, Z., Zhao, R., Mangal, R., Fredrikson, M., Jia, L., & Păsăreanu, C. S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695297 · Cited by 22
Self-citationWill multimodal large language models ever achieve deep understanding of the world?
Farkaš, I., Vavrečka, M., & Wermter, S. (2025) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2025.1683133 · Cited by 6
Bootstrapping Cognitive Agents with a Large Language Model
Zhu, F., & Simmons, R. G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27822 · Cited by 11
The Bias that Lies Beneath: Qualitative Uncovering of Stereotypes in Large Language Models
Babonnaud, W., Delouche, E., & Lahlouh, M. (2024) · Linköping electronic conference proceedings · doi:10.3384/ecp208022 · Cited by 9
Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
Hossain, M. Z., & Imteaj, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825099 · Cited by 18
Nirmal, S., Patil, P. R., & Shinde, S. B. (2025) · e-Prime - Advances in Electrical Engineering Electronics and Energy · doi:10.1016/j.prime.2025.100909 · Cited by 6
Artificial intelligence, adversarial attacks, and ocular warfare
Balas, M., Wong, D. T., & Arshinoff, S. A. (2024) · AJO International · doi:10.1016/j.ajoint.2024.100062 · Cited by 6
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.09024 · Cited by 11
Self-citationPan-indexicality and prompt: developing a teaching model for AI-mediated academic writing
Zhu, J. J., & Duan, C. (2025) · Language and Semiotic Studies · doi:10.1515/lass-2025-0008 · Cited by 5
Cybersecurity challenges and opportunities of machine learning-based artificial intelligence
Czaja, P., Gdowski, B., Niemiec, M., Mees, W., Stoianov, N., Votis, K., Kharchenko, V. S., Katos, V., & Merialdo, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11604-9 · Cited by 5
Chen, R., Cui, S., Huang, X., Pan, C., Huang, V. S.-J., Zhang, Q., Ouyang, X., Zhang, Z., Wang, H., & Huang, M. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754561 · Cited by 5
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 40
The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
Chochlakis, G., Potamianos, A., Lerman, K., & Narayanan, S. S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00041 · Cited by 11
ArtPerception: ASCII art-based jailbreak on LLMs with recognition pre-test
Yang, G.-Y., Cheng, T.-Y., Teng, Y.-W., Wang, F., & Yeh, K. (2025) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2025.104356 · Cited by 5
Towards a Conversational Ethics of Large Language Models
Kempt, H., Lavie, A., & Nagel, S. K. (2024) · American Philosophical Quarterly · doi:10.5406/21521123.61.4.04 · Cited by 7
Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally
Al-Maliki, S., Qayyum, A., Ali, H. M., Abdallah, M. M., Qadir, J., Hoang, D. T., Niyato, D. T., & Al‐Fuqaha, A. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3383407 · Cited by 10
Attacking Vision-Language Computer Agents via Pop-ups
Zhang, Y., Yu, T., & Yang, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.411 · Cited by 4
Politicians vs ChatGPT: A study of presuppositions in French and Italian political communication
Garassino, D., Masia, V., Brocca, N., & Delorme Benites, A. (2024) · AI-Linguistica. Linguistic Studies on AI-Generated Texts and Discourses · doi:10.62408/ai-ling.v1i1.5 · Cited by 4
Ruiu, D. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_24 · Cited by 4
Artificial Intelligence System Risk Management Methodology Based on Generalized Blueprints
Bogdanov, D., Etti, P., Kamm, L., & Stomakhin, F. (2024) · International Conference on Cyber Conflict · doi:10.23919/cycon62501.2024.10685644 · Cited by 4
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4
Safety of LLM-based AI chatbots for young consumers in purchase decisions
Tan, A. H. T., & Leo, K. C. (2025) · Young Consumers Insight and Ideas for Responsible Marketers · doi:10.1108/yc-10-2024-2277 · Cited by 8
Large Language Models in jihadist terrorism and crimes
Puczyńska, J., Podhajski, M., Wojtasik, K., & Michalak, T. (2024) · Terroryzm · doi:10.4467/27204383ter.24.012.19400 · Cited by 7
Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Framework
Ma, B., Guo, H., Luo, Z., & Duan, R. (2025) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w65791.2025.00074 · Cited by 6
Acosta-Bermejo, R., Terrazas-Chavez, J. A., & Anaya, E. A. (2025) · Applied Sciences · doi:10.3390/app15179252 · Cited by 4
The power of generative AI in cybersecurity: Opportunities and challenges
Wen, S. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/48/20241095 · Cited by 5
Xiao, B., Yin, Z., & Shan, Z. (2023) · arXiv · doi:10.48550/arxiv.2311.06957 · Cited by 4
“Additionally, while the LLM used targeted removal of harmful speech during training [10, 58, 59], to ensure diverse generative agent populations resembling small town residents, we de-“harmless” the LLM [60].”
MethodAI Control: Improving Safety Despite Intentional Subversion
Greenblatt, R., Shlegeris, B., Sachan, K., & Roger, F. (2023) · arXiv · doi:10.48550/arxiv.2312.06942 · Cited by 3
“For more on this choice, see Appendix D.”
BackgroundJin, K., Xiong, Y., Lou, S., & Yu, Z. X. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11710-0 · Cited by 5
Mo, Y., Tang, M., Lin, R., Zhou, B., & Li, X. (2025) · IEEE Access · doi:10.1109/access.2025.3582519 · Cited by 5
Misalignment or misuse? The AGI alignment tradeoff
Hellrigel-Holderbaum, M., & Dung, L. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02403-y · Cited by 6
CLEVA: Chinese Language Models EVAluation Platform
Li, Y., Zhao, J., Zheng, D., Hu, Z.-Y., Chen, Z., Su, X., Huang, Y., Huang, S., Lin, D., Lyu, M. R.-T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.17 · Cited by 4
A Causal Framework for AI Regulation and AuditingPreprint
Sharkey, L., Ghuidhir, C. N., Braun, D., Scheurer, J., Balesni, M., Bushnaq, L., Stix, C., & Hobbhahn, M. (2024) · Preprints.org · doi:10.20944/preprints202401.1424.v1 · Cited by 9
RNA-DCGen: Dual Constrained RNA Sequence Generation with LLM-AttackPreprint
Shahgir, H. S., Ratul, M. R. Z., Tahmid, M. T., Sayeed, K. S., & Rahman, A. H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.23.614570 · Cited by 3
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
“…exposing private data through surveillance or by mounting cyber-offensive operations against rival agents—potentially weaponizing adversarial attacks, jailbreaks, and resource denial at scale (Ellsberg, 1968; Harrenstein, 2007; Zou, 2023; Gleave et al., 2020; Yamin, 2021; Brundage et al., 2018b).”
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.10019 · Cited by 2
“More recent studies utilized red-teaming such as adversarial (Zou et al., 2023) and jailbreak attacks (Wei et al., 2023; Yuan et al., 2023; Deng et al.; Yi et al., 2023), to evaluate (Bhardwaj and Poria, 2023) safety of LLMs.”
BackgroundCommercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M. (2025) · arXiv · doi:10.48550/arxiv.2502.08586 · Cited by 2
“Many cutting-edge attack algorithms involve gradient-based or transfer attacks (Zou et al., 2023), but these are impractical for LLM agents since they often require white-box access for computing gradients or require a very large number of model calls.”
Riepin, P. (2024) · Teisė · doi:10.15388/teise.2024.130.10 · Cited by 2
Navigating the Risks: A Review of Safety Issues in Large Language Models
Wang, H., Li, Y., Wang, Y., Liu, P., & Li, P. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00020 · Cited by 2
Computing-Model and Computing-Hardware Selection for ICT Under Societal and Judicial Constraints
Bock, Y. N., Boche, H., Fitzek, F. H. P., & Gitta Kutyniok, B. (2024) · IEEE Access · doi:10.1109/access.2024.3519979 · Cited by 2
DS-GCG: Enhancing LLM Jailbreaks with Token Suppression and Induction Dual-Strategy
Tang, X., Yang, X., Yao, Z., Wen, J., Zhou, X., Han, J., & Hu, S. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033375 · Cited by 2
Yi, X., Li, Y., Shi, D., Wang, L., Wang, X., & He, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129101 · Cited by 8
Red Teaming for Multimodal Large Language Models: A SurveyPreprint
Mahato, M., Kumar, A., Singh, K., Kukreja, B., & Nabi, J. (2024) · doi:10.36227/techrxiv.170629758.87975697/v1 · Cited by 2
AI Outputs Are Not Protected SpeechPreprint
Salib, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4687558 · Cited by 2
Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offsPreprint
Fonseca, J., Bell, A. S., & Stoyanovich, J. (2025) · Qeios · doi:10.32388/zcxbc8 · Cited by 2
Manipulating Large Language Models to Increase Product Visibility
Kumar, A., & Lakkaraju, H. (2024) · arXiv · doi:10.48550/arxiv.2404.07981 · Cited by 2
Natural Emergent Misalignment from Reward Hacking in Production RL
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.18397 · Cited by 1
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., & Chaudhuri, K. (2025) · arXiv · doi:10.48550/arxiv.2504.18575 · Cited by 1
AdvAgent: Controllable Blackbox Red-teaming on Web Agents
Xu, C., Kang, M., Zhang, J., Liao, Z., Mo, L., Yuan, M., Sun, H., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2410.17401 · Cited by 1
Private Information Leakage in LLMs
Buesser, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_7 · Cited by 1
Enhancing Security Awareness and Education for LLMs
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_18 · Cited by 1
Artificial Intelligence Technologies in Official Statistics: Use Cases and Risks
Башина, О. Э., Матраева, Л. В., & Vasiutina, E. S. (2025) · Вопросы статистики · doi:10.34023/2313-6383-2025-32-2-5-14 · Cited by 1
The superalignment of superhuman intelligence with large language models
Huang, M., Wang, Y., Cui, S., Ke, P., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4348-6 · Cited by 3
Xu, S., Luo, F., Li, S., Fan, M., & Sun, Z. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2500421 · Cited by 1
Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Models
Ke, S.-W., Lai, G.-Y., Fang, G.-L., & Kao, H.-Y. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0887-7_33 · Cited by 2
Operationalizing Responsible AI Policies with LLMs: an End-to-End Monitoring Prototype
Zieliński, P., Kelly, M., Barreto, R. D. C., & Tahaei, M. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799675 · Cited by 2
Small but Dangerous: Evaluating and Mitigating Jailbreak Vulnerabilities in Small Language Models
Piano, L., Battistin, C., den Abeele, J. V., & Pompianu, L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19099-4_35 · Cited by 1
Uhm, J., Kim, M., Polychronakis, M., & Koo, H. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797116 · Cited by 1
Metamorphic Stress Testing of Vision-Language Components for Time-Critical Autonomous Systems
Chhetri, A., & Liu, Y. (2026) · doi:10.1145/3843782.3844660 · Cited by 1
Unnatural language processing: How do language models handle machine-generated prompts?
Kervadec, C., Franzon, F., & Baroni, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.959 · Cited by 1
Generative AI for Trustworthy, Open, and Equitable Scholarship
Bourg, C., Kriegsman, S., Lindsay, N., Sardis, H., Stalberg, E., & Altman, M. (2024) · doi:10.21428/e4baedd9.567bfd15 · Cited by 1
AI is vulnerable to attack. Can it ever be used safely?
Makin, S. J. (2024) · Nature · doi:10.1038/d41586-024-02419-0 · Cited by 1
Li, X., Shen, Q., & Kawaguchi, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01175 · Cited by 6
Frustratingly Easy Jailbreak of Large Language Models via Output Prefix AttacksPreprint
Wang, Y., Chen, M., Peng, N., & Chang, K.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-4385503/v1 · Cited by 1
A Comprehensive Review of Security and Privacy Issues in Large Language ModelsPreprint
Inamdar, A. S., & Eswaran, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4825655 · Cited by 2
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 4
Yi, X., Li, Y., Wang, L., Wang, X., & He, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5169050 · Cited by 1
Rules Created by Symbolic Systems Cannot Constrain a Learning System v3, 2025-05-16Preprint
LIN, S.-W., Xu, R., Li, X., & Xu, W. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5256795 · Cited by 1
Chen, S. (2026) · arXiv · doi:10.48550/arxiv.2606.22528 · Cited by 21
“Pinning and the system-channel policy both hold at 0% / 0% , but the system channel is available only when the operator controls the system prompt—not for memory-or tool-delivered governance.”
CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents
Hossain, S. M. A., Shayoni, R. K., & Morol, M. K. (2026) · arXiv · doi:10.48550/arxiv.2609.02265 · Cited by 3
“Safety alignment is shallow in the token dimension (Qi et al., 2025), breakable by optimized suffixes (Zou et al., 2023), and fragile under fine-tuning (Qi et al., 2024), which argues for keeping the constraint outside anything a user can revise (Bai et al., 2022), and our selector is a…”
Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats
Thota, P., Lei, Y., Thangaraj, S., Jonnalagadda, S. R., & Nilizadeh, S. (2026) · arXiv · doi:10.48550/arxiv.2606.26377
“…[39] containing real-world adversarial jailbreak prompts using fictional framing, persona assignment, and researcher impersonation; AdvBench [40] with 520 harmful instruction-target pairs spanning hacking, weapons, and identity theft; StrongReject [41] with 313 forbidden prompts across…”
Ctrl-Z: Controlling AI Agents via Resampling
Bhatt, A., Rushing, C., Kaufman, A., Tracy, T., Georgiev, V., Matolcsi, D., Khan, A., & Shlegeris, B. (2025) · arXiv · doi:10.48550/arxiv.2504.10374
“The rapid rise of Large Language Models (LLMs) and autonomous AI agents (Achiam et al., 2023; Xi et al., 2023) opens new opportunities and challenges in ensuring their safe operation (Chan et al., 2023).”
Efficient LLM Safety Evaluation through Multi-Agent Debate
Lin, D., Shen, G., Yang, Z., Liu, T., Zhao, D., & Zeng, Y. (2025) · arXiv · doi:10.48550/arxiv.2511.06396 · Cited by 8
A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
Schwinn, L., Ladenburger, M., Beyer, T., Mofakhami, M., Gidel, G., & Günnemann, S. (2026) · arXiv · doi:10.48550/arxiv.2603.06594 · Cited by 19
“As red-teaming efforts scale to identify vulnerabilities, manual human evaluation has become prohibitively expensive, leading to a reliance on ”LLM-as-a-Judge” frameworks (Zou et al., 2023).”
MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
Zhang, X., Zheng, Y., Xu, Z., Zhou, K., Shen, B., Ou, H., Zhang, T., & Lam, K.-Y. (2026) · arXiv · doi:10.48550/arxiv.2605.26154
Roesner, F., & Kohno, T. (2026) · arXiv · doi:10.48550/arxiv.2609.17817 · Cited by 2
Multi-Agent Systems Execute Arbitrary Malicious Code
Triedman, H., Jha, R., & Shmatikov, V. (2025) · arXiv · doi:10.48550/arxiv.2503.12188 · Cited by 52
Some Useful Things to Know When Combining IR and NLP: The Easy, the Hard and the Ugly
Alonso, O., & Church, K. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3636452
Shan, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4733550
Guerraoui, R., & Pinot, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_20
OQA : A question-answering dataset on orthodontic literaturePreprint
Rousseau, M., Zouaq, A., & Huynh, N. T. (2024) · medRxiv · doi:10.1101/2024.07.05.24309412
Not What I was Trained for – Out-of-Distribution-Tests for Interactive AIs
Severin, B., Werger, O., & Hesenius, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-59235-5_12
Hamed, M., Kasiem, N., Abdou, M., Aslan, H. K., Khoriba, G. A., & Arafa, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4816315/v1
Enhancing Adversarial Attacks: The Similar Target Method
Zhang, S., Wang, Z., Zhou, Z., Liu, J., & Chen, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649974
Self-citationLiu, Z., Li, G., Shi, B., Zhang, X., Ge, J., Wu, Y., & Lyu, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650172 · Cited by 1
From Large Language Models to Generative AI Systems
Tomczak, J. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-64087-2_11
Deceptively simple: An outsider's perspective on natural language processing
KhudaBukhsh, A. R. (2024) · AI Magazine · doi:10.1002/aaai.12204
Metadata
- Authors
- Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M.
- Affiliations
- Carnegie Mellon University (US)
- Google DeepMind (United Kingdom) (GB)
- Robert Bosch (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.15043
- OpenAlex
- W4385374425
- Cited by
- 197
- References
- 0
- Citations by year
- 2026: 31 · 2025: 61 · 2024: 89 · 2023: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1943016)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- Multidisciplinary University Research Initiative (W911NF-21-1-0317)
- Army Research Office (W911NF-21-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- jailbreak attacks, voice assistants, audio modulation, black-box evaluation, audio-to-audio translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
