Universal and Transferable Adversarial Attacks on Aligned Language Models
Open accessAbstract
Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In black-box settings, the attacker has no access to model parameters or gradients, relying solely on agent responses, which makes gradient-based optimization techniques ineffective.
“The attacker has no access to the model's parameters or gradients, relying solely on the agent's responses to adversarial inputs, which makes gradient-based optimization techniques (Zou et al., 2023) ineffective.”
From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Zou et al., 2023
Section 2.2 describes Zou et al.’s gradient-based optimization method, and the Introduction reports that its prompts successfully attacked GPT-3.5 and GPT-4 through transfer. The claim that gradient-based techniques are ineffective in black-box settings overlooks this distinction: the paper uses gradients from accessible models to generate attacks that transfer to proprietary targets. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredExisting LLM red teaming focuses on red teaming the model by generating inputs that lead to undesirable behavior, absent monitoring or other countermeasures.
“Existing LLM red teaming focuses on red teaming the model by generating inputs that lead to undesirable behavior, absent monitoring or other countermeasures (Perez et al., 2022; Wei et al., 2023; Zou et al., 2023).”
From AI Control: Improving Safety Despite Intentional Subversion (2023) · cites this paper as Zou et al., 2023
Section 3, “Experimental Results: Direct and Transfer Attacks” (page unavailable), describes generating attacks to elicit harmful model output and calls the task a red-teaming exercise. But the claim’s “absent monitoring or other countermeasures” qualification is not fully supported: Section 3.3 discusses an input content filter on Claude and adapting attacks to evade it. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAI security has remained largely model-centric, focused on single-agent attack surfaces and adversarial samples.
“AI security, by contrast, has remained largely model-centric, focused on single-agent attack surfaces and adversarial samples (Biggio et al., 2012;Szegedy et al., 2014;Zou et al., 2023).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
The Introduction describes extensive work on adversarial attacks against machine-learning models, and “Adversarial examples & transferability” calls adversarial examples a subject of extensive research. The paper also tests attacks on individual models (§3.1), but it does not characterize AI security as a whole as largely model-centric or establish a focus on single-agent attack surfaces. Page numbers are unavailable in our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAdvanced AI promises to lower the cost and broaden the scope of coercion and extortion-whether by exposing private data through surveillance or by mounting cyber-offensive operations against rival agents-potentially weaponizing adversarial attacks, jailbreaks, and resource denial at scale.
“Moreover, advanced AI promises to lower the cost and broaden the scope of coercion and extortion -whether by exposing private data through surveillance or by mounting cyber-offensive operations against rival agents -potentially weaponizing adversarial attacks, jailbreaks, and resource denial at scale (Ellsberg, 1968;Harrenstein et al., 2007;Zou et al., 2023;Gleave et al., 2020;Brundage et al., 202”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Zou et al., 2023
The Introduction and §3.3 Discussion describe jailbreaks and automated adversarial attacks on language models, and the transfer results show attacks eliciting harmful responses across multiple models. The Ethics and Broader Impact section warns that these techniques could help generate harmful content. The paper does not establish the broader claim that AI lowers the cost of coercion or extortion through surveillance, cyber-offensive operations, or resource denial at scale. (LLM, conf=0.85)
Experimental — accuracy not yet measuredOpenAI appeared to patch its system to prevent adversarial attacks with specific suffixes discovered in Zou, Wang, et al. (2023), but the underlying vulnerability may still persist.
“For instance, OpenAI appeared to patch its system to prevent adversarial attacks with specific suffixes discovered in Zou, Wang, et al. (2023), but the underlying vulnerability may still persist.”
From Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools (2025) · cites this paper as Zou, Wang, et al. (2023)
The Responsible Disclosure section says the authors shared preliminary results with OpenAI. Section 6 (Ethics and Broader Impact) predicts that the examples shown will likely cease to function and says it remains unclear how the underlying challenge can be addressed. It does not report that OpenAI patched its system or that particular suffixes had stopped working. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Jailbreaks aim to escape the safety behavior trained into an LLM and often do not specifically conflict with a model's previous instructions.
“Jailbreaks Another well-known attack vector is jailbreak, which specifically aims to escape the safety behavior that is trained into an LLM.As such, they often don't specifically conflict with a model's previous instructions.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Zou et al., 2023
The Introduction (page unavailable) describes jailbreaks as prompts that cause aligned LLMs to generate objectionable content, supporting the claim that they aim to escape trained safety behavior. Neither that passage nor Section 2.1 (page unavailable) establishes that jailbreaks often do not specifically conflict with a model’s previous instructions. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSafety alignment is shallow in the token dimension, breakable by optimized suffixes, and fragile under fine-tuning.
“Safety alignment is shallow in the token dimension (Qi et al., 2025), breakable by optimized suffixes (Zou et al., 2023), and fragile under fine-tuning (Qi et al., 2024)”
From CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents (2026) · cites this paper as Zou et al., 2023
LLMs are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs.
“In addition, they are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs (Kang et al., 2023; Wei et al., 2023; Zou et al., 2023).”
From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Zou et al., 2023
Such failures are surfaced by automated red-teaming and standardized refusal benchmarks and intensify under optimization, from transferable adversarial suffixes to long-context many-shot prompting.
“Such failures are surfaced by automated red-teaming and standardized refusal benchmarks (Perez et al. 2022; Mazeika et al. 2024) and intensify under optimization, from transferable adversarial suffixes (Zou et al. 2023) to long-context many-shot prompting (Anil et al. 2024).”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Zou et al. 2023
This work is the first to repurpose jailbreaking techniques for detecting LLM-generated peer reviews.
“our work is the first to repurpose them for detecting LLM-generated peer reviews.”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The Greedy Coordinate Gradient (GCG) algorithm was originally developed for jailbreaking LLMs.
“the Greedy Coordinate Gradient (GCG) algorithm, originally developed for jailbreaking LLMs [19]”
From Detecting LLM-generated peer reviews (2025) · cites this paper as [19]
The study does not explore more advanced jailbreaking techniques, such as optimizing the string inside the pop-ups or making the pop-ups more persuasive.
“(ii) We do not explore more advanced jailbreaking techniques, such as optimizing the string inside the pop-ups (Zou et al., 2023) or making our pop-ups more persuasive (Zeng et al., 2024), but focus more on the high-level design of the adversarial pop-ups and the contribution analysis of different components.”
From Attacking Vision-Language Computer Agents via Pop-ups (2025) · cites this paper as Zou et al., 2023
Current LLM security research predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks or extracting memorized training data.
“Current research in LLM security predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks (Zou et al., 2023;Liu et al., 2024) or on extracting memorized training data from a trained model (Nasr et al., 2023).”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
Many cutting-edge attack algorithms involve gradient-based or transfer attacks, but these are impractical for LLM agents since they often require white-box access for computing gradients or a very large number of model calls.
“Many cutting-edge attack algorithms involve gradient-based or transfer attacks (Zou et al., 2023), but these are impractical for LLM agents since they often require white-box access for computing gradients or require a very large number of model calls.”
From Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks (2025) · cites this paper as Zou et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Hadi Abdullah, Aditya Karlekar, Vincent Bindschaedler, and Patrick Traynor. Demystifying limited adversarial transferability in automatic speech recognition systems. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=l5aSHXi8jG5.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
- Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning. PMLR, 2019.
- Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335, 2022.
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning {ai} with shared human values. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=dNy_RKzJacY.
- Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human prefer- ences. In International Conference on Machine Learning, pages 17506-17533. PMLR, 2023.
- Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning. PMLR, 2021.
- Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In 2023 IEEE Symposium on Security and Privacy (SP), 2023.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.
- Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based con- strained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9551, 2022.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113 · see citing works
“Such general addendums that can be added to any prompt to improve the results are sometimes referred to as a “universal and transferrable prompt suffix”, which is frequently employed as a method to successfully jailbreak an LLM (Zou et al., 2023).”
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445 · see citing works
Generative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293 · see citing works
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · see citing works
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243 · see citing works
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Ferrag, M. A., Friha, O., Kantarcı, B., Tihanyi, N., Cordeiro, L. C., Debbah, M. A., Hamouda, D., Al-Hawawreh, M. S., & Choo, K. R. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3317242 · Cited by 170 · see citing works
Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182 · see citing works
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.40 · Cited by 130 · see citing works
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 128 · see citing works
Addressing 6 challenges in generative AI for digital health: A scoping review
Templin, T., Perez, M. W., Sylvia, S., Leek, J. T., & Sinnott-Armstrong, N. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000503 · Cited by 115 · see citing works
Bobula, M. (2024) · Journal of Learning Development in Higher Education · doi:10.47408/jldhe.vi30.1137 · Cited by 113 · see citing works
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147 · see citing works
Roberts, J., Baker, M., & Andrew, J. (2024) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2024.102722 · Cited by 100 · see citing works
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.130 · Cited by 85 · see citing works
Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00685 · Cited by 100 · see citing works
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A. A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659037 · Cited by 104 · see citing works
On the Adversarial Robustness of Multi-Modal Foundation Models
Schlarmann, C., & Hein, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00395 · Cited by 78 · see citing works
MMA-Diffusion: MultiModal Attack on Diffusion Models
Yang, Y., Gao, R., Wang, X., Ho, T.-Y., Xu, N., & Xu, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00739 · Cited by 69 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3377445 · Cited by 62 · see citing works
Medical large language models are susceptible to targeted misinformation attacks
Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01282-7 · Cited by 62 · see citing works
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57 · see citing works
Self-citation<scp>PLeak:</scp> Prompt Leaking Attacks against Large Language Model Applications
Hui, B., Yuan, H., Gong, N., Burlina, P., & Cao, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670370 · Cited by 55 · see citing works
On the Robustness of Large Multimodal Models Against Image Adversarial Attacks
Cui, X., Aparcedo, A., Jang, Y. K., & Lim, S.-N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02325 · Cited by 60 · see citing works
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 49 · see citing works
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 70 · see citing works
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Gyevnár, B., & Kasirzadeh, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01020-y · Cited by 40 · see citing works
Shah, S. B., Thapa, S., Acharya, A., Rauniyar, K., Poudel, S., Jain, S., Masood, A., & Naseem, U. (2024) · IEEE Access · doi:10.1109/access.2024.3406644 · Cited by 41 · see citing works
LLM-PBE: Assessing Data Privacy in Large Language Models
Li, Q., Hong, J., Xie, C., Tan, J., Xin, R., Hou, J., Yin, X., Wang, Z., Hendrycks, D., Wang, Z., et al. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681994 · Cited by 42 · see citing works
Self-citationLarge Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 47 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“…content have drawn significant attention from the community (Bang et al., 2023; Alkaissi & McFarlane, 2023; Zheng et al., 2023; Shi et al., 2023; Carlini et al., 2021; Huang et al., 2022; Shao et al., 2023; Li et al., 2023; Wei et al., 2023; Zhou et al., 2023b; Zou et al., 2023, inter alia ).”
BackgroundDeliberative Alignment: Reasoning Enables Safer Language Models
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15159 · Cited by 106 · see citing works
Three Challenges to Secure AI Systems in the Context of AI Regulations
Hamon, R., Junklewitz, H., Garrido, J. S., & Sánchez, I. (2024) · IEEE Access · doi:10.1109/access.2024.3391021 · Cited by 34 · see citing works
PromptShield: Deployable Detection for Prompt Injection Attacks
Jacob, D., Alzahrani, H., Hu, Z., Alomair, B., & Wagner, D. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726501 · Cited by 35 · see citing works
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31 · see citing works
Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
Ying, Z., Liu, A., Zhang, T., Yu, Z., Liang, S., Liu, X., & Tao, D. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3583249 · Cited by 29 · see citing works
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
“Originally proposed for jailbreaking, Greedy Coordinate Gradient (GCG) [64] has been adapted to IPI by generating affirmative prefixes containing adversarial strings that induce malicious outputs from the agent [65].”
Application of generative language models to orthopaedic practice
Caterson, J., Ambler, O., Cereceda‐Monteoliva, N., Horner, M., Jones, A., & Poacher, A. T. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-076484 · Cited by 27 · see citing works
Angel, M. C., Rinehart, J. B., Canneson, M. P., & Baldi, P. (2024) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006892 · Cited by 26 · see citing works
METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Hyun, S., Guo, M., & Babar, M. A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00019 · Cited by 26 · see citing works
Can AI have common sense? Finding out will be key to achieving machine intelligence
Kejriwal, M., Santos, H., Mulvehill, A. M., Shen, K., McGuinness, D. L., & Lieberman, H. A. (2024) · Nature · doi:10.1038/d41586-024-03262-z · Cited by 26 · see citing works
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
Pasquini, D., Strohmeier, M., & Troncoso, C. (2024) · AISec@CCS · doi:10.1145/3689932.3694764 · Cited by 27 · see citing works
Maximising Large Language Model Utility in Cardiovascular Care: A Practical Guide
Nolin-Lapalme, A., Thériault-Lauzier, P., Corbin, D., Tastet, O., Sharma, A., Hussin, J., Kadoury, S., Jiang, R., Krahn, A. D., Gallo, R., et al. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.05.024 · Cited by 28 · see citing works
Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
Hu, Z., Wang, C., Shu, Y., Paik, H.-Y., & Zhu, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671932 · Cited by 27 · see citing works
Red teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 25 · see citing works
Tsamados, A., Floridi, L., & Taddeo, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4578165 · Cited by 22 · see citing works
Unveiling Privacy Risks in LLM Agent Memory
Wang, B., He, W., Zeng, S., Xiang, Z., Xing, Y., Tang, J., & He, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1227 · Cited by 24 · see citing works
How do people react to political bias in generative artificial intelligence (AI)?
Messer, U. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100108 · Cited by 24 · see citing works
LangTest: A comprehensive evaluation library for custom LLM and NLP models
Nazir, A., Chakravarthy, T. K., Cecchini, D., Khajuria, R., Sharma, P., Mirik, A. T., Kocaman, V., & Talby, D. (2024) · Software Impacts · doi:10.1016/j.simpa.2024.100619 · Cited by 22 · see citing works
Artificial intelligence security and privacy: a survey
He, X., Xu, G., Han, X., Wang, Q., Zhao, L., Shen, C., Lin, C., Zhao, Z., Li, Q., Yang, L., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4388-5 · Cited by 23 · see citing works
Self-citationDoes ChatGPT4 have a dialogical self?: A Bakhtinian perspective
Matusov, E., Chat, G. P. T., Smith, M. P., & Shugurova, O. (2023) · Culture & Psychology · doi:10.1177/1354067x231219454 · Cited by 20 · see citing works
Fine-Tuning LLMs with Medical Data: Can Safety Be Ensured?
Kim, M., Kim, Y., Kang, H. J., Seo, H., Choi, H., Han, J., Kee, G., Park, S., Ko, S., Jung, H., et al. (2024) · NEJM AI · doi:10.1056/aics2400390 · Cited by 20 · see citing works
Adversarial prompt and fine-tuning attacks threaten medical large language models
Yang, Y., Jin, Q., Huang, F., & Lu, Z. (2025) · Nature Communications · doi:10.1038/s41467-025-64062-1 · Cited by 31 · see citing works
Defense against prompt injection attack by leveraging attack techniques
Chen, Y., Li, H., Zheng, Z., Wu, D., Song, Y., & Hooi, B. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.897 · Cited by 19 · see citing works
Adversarial robust image processing in medical digital twin
Shamshiri, S., Liu, H., & Sohn, I. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102728 · Cited by 22 · see citing works
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Yip, D. W., Esmradi, A., & Chan, C. F. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487667 · Cited by 19 · see citing works
Detecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14 · see citing works
“Although such jailbreaking techniques have been used in the broader LLM safety literature [ZWC + 23, WHS24] to elicit harmful outputs (e.g., instructions for building a bomb), our work is the first to repurpose them for detecting LLM-generated peer reviews.”
InfluentialThe Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14 · see citing works
“There are a myriad of attack variants (Schulhoff et al., 2023; Zou et al., 2023; Wei et al., 2023) that can allow adversaries to perform malicious tasks such as generating spam, misinformation, or producing pornographic content.”
BackgroundChu, K.-F., Yuan, H., Yuan, J., Guo, W., Balta‐Ozkan, N., & Li, S. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3427655 · Cited by 14 · see citing works
Huang, X., Wang, X., Zhang, H., Zhu, Y., Xi, J., An, J., Wang, H., Liang, H., & Pan, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32396 · Cited by 14 · see citing works
Intentional Biases in LLM Responses
Badyal, N., Jacoby, D., & Coady, Y. (2023) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon59035.2023.10316060 · Cited by 13 · see citing works
On Large Language Models’ Resilience to Coercive Interrogation
Zhang, Z., Shen, G., Tao, G., Cheng, S., & Zhang, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00208 · Cited by 14 · see citing works
A comparative analysis on using GPT and BERT for automated vulnerability scoring
Mirtaheri, S. L., Pugliese, A., Movahedkor, N., & Shahbazian, R. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200515 · Cited by 13 · see citing works
Lessons from complex systems science for AI governance
Kolt, N., Shur‐Ofry, M., & Reuven Cohen, C. (2025) · Patterns · doi:10.1016/j.patter.2025.101341 · Cited by 22 · see citing works
Large Language Models and Security
Michele Bezzi, A. (2024) · IEEE Security & Privacy · doi:10.1109/msec.2023.3345568 · Cited by 12 · see citing works
Silent Guardian: Protecting Text From Malicious Exploitation by Large Language Models
Zhao, J., Chen, K., Yuan, X., Qi, Y., Zhang, W., & Yu, N. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3455775 · Cited by 12 · see citing works
On the Societal Impact of Open Foundation Models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07918 · Cited by 11 · see citing works
The EU AI Act: A pioneering effort to regulate frontier AI?
Graells, G. B., Salinas, C., Tinoco, R., & Sevilla, J. (2024) · INTELIGENCIA ARTIFICIAL · doi:10.4114/intartif.vol27iss73pp55-64 · Cited by 11 · see citing works
Large Language Models and Computer Security
Iyengar, A., & Kundu, A. (2023) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa58951.2023.00045 · Cited by 12 · see citing works
Limitations of Large Language Models
Sanu, E., Amudaa, T. K., Bhat, P., Dinesh, G., Chate, A. U. K., & Ramakanth Kumar, P. (2024) · International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS) · doi:10.1109/csitss64042.2024.10817070 · Cited by 12 · see citing works
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
Liu, Z., Gan, C. G., Wang, J., Zhang, Y., Bo, Z., Sun, M., Chen, H., & Zhang, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714816 · Cited by 11 · see citing works
Breaking Bad: Unraveling Influences and Risks of User Inputs to ChatGPT for Game Story Generation
Taveekitworachai, P., Abdullah, F., Gursesli, M. C., Dewantoro, M. F., Chen, S., Lanatà, A., Guazzini, A., & Thawonmas, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47658-7_27 · Cited by 11 · see citing works
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 10 · see citing works
“However, current attacks targeting LLMs, such as jailbreaking [10, 42] during testing and backdooring in-context learning [31], cannot effectively attack LLM agents with RAG.”
InfluentialEvil Geniuses: Delving into the Safety of LLM-based Agents
Tian, Y., Yang, X., Zhang, J., Dong, Y., & Su, H. (2023) · arXiv · doi:10.48550/arxiv.2311.11855 · Cited by 10 · see citing works
Vulnerabilities Introduced by LLMs Through Code Suggestions
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_9 · Cited by 10 · see citing works
ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models
Jin, H., Zhang, C., Shi, S., Lou, W., & Hou, Y. T. (2024) · IEEE Conference on Communications and Network Security · doi:10.1109/cns62487.2024.10735575 · Cited by 11 · see citing works
Shang, S., Zhao, X., Yao, Z., Yao, Y., Su, L., Fan, Z., Zhang, X., & Jiang, Z. (2024) · The Computer Journal · doi:10.1093/comjnl/bxae124 · Cited by 15 · see citing works
Omri, S., Abdelkader, M., & Hamdi, M. (2025) · Journal of Advances in Information Technology · doi:10.12720/jait.16.2.243-250 · Cited by 11 · see citing works
A Lifecycle-Oriented Survey of Emerging Threats and Vulnerabilities in Large Language Models
de Maio, C., Di Gisi, M., Fenza, G., Gallo, M., & Loia, V. (2025) · IEEE Access · doi:10.1109/access.2025.3619764 · Cited by 10 · see citing works
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9 · see citing works
Defending Large Language Models Against Jailbreak Attacks Through Chain of Thought Prompting
Cao, Y., Gu, N., Shen, X., Yang, D., & Zhang, X. (2024) · International Conference on Networking and Network Applications (NaNA) · doi:10.1109/nana63151.2024.00028 · Cited by 9 · see citing works
Rudinskiy, M., Morone, D., & Molinari, M. (2024) · Traffic · doi:10.1111/tra.12957 · Cited by 10 · see citing works
Eliminating Backdoors in Neural Code Models for Secure Code Understanding
Sun, W., Chen, Y., Fang, C., Feng, Y., Xiao, Y., Guo, A., Zhang, Q., Chen, Z., Xu, B., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715782 · Cited by 11 · see citing works
Towards algorithms and models that we can trust: A theoretical perspective
Oneto, L., Ridella, S., & Anguita, D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127798 · Cited by 9 · see citing works
Ranking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8 · see citing works
“Our ethical considerations are similar to those in established jailbreaking attacks (Zou et al., 2023).”
BackgroundLeverage Large Language Models For Enhanced Aviation Safety
Fox, K. L., Niewoehner, K. R., Rahmes, M. D., Wong, J., & Razdan, R. (2024) · Integrated Communications, Navigation and Surveillance Conference (ICNS) · doi:10.1109/icns60906.2024.10550651 · Cited by 8 · see citing works
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 10 · see citing works
Cong, T., Ran, D., Liu, Z., He, X., Liu, J., Gong, Y., Li, Q. L., Wang, A., & Wang, X. (2023) · doi:10.1145/3689217.3690614 · Cited by 11 · see citing works
LLM-Sentry: A Model-Agnostic Human-in-the-Loop Framework for Securing Large Language Models
Irtiza, S., Akbar, K. A., Yasmeen, A., Khan, L. R., Daescu, O., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00036 · Cited by 8 · see citing works
- Authors
- Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M.
- Affiliations
- Carnegie Mellon University (US)
- Google DeepMind (United Kingdom) (GB)
- Robert Bosch (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.15043
- OpenAlex
- W4385374425
- Cited by
- 197
- References
- 0
- Citations by year
- 2026: 31 · 2025: 61 · 2024: 89 · 2023: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1943016)
- Defense Advanced Research Projects Agency (FA8750, FA8750-15-2-0277)
- Multidisciplinary University Research Initiative (W911NF-21-1-0317)
- Army Research Office (W911NF-21-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- jailbreak attacks, voice assistants, audio modulation, black-box evaluation, audio-to-audio translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
