Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks
Open accessAbstract
Recent explorations with commercial Large Language Models (LLMs) have shown that non-expert users can jailbreak LLMs by simply manipulating their prompts; resulting in degenerate output behavior, privacy and security breaches, offensive outputs, and violations of content regulator policies.Limited studies have been conducted to formalize and analyze these attacks and their mitigations.We bridge this gap by proposing a formalism and a taxonomy of known (and possible) jailbreaks.We survey existing jailbreak methods and their effectiveness on open-source and commercial LLMs (such as GPT-based models, OPT, BLOOM, and FLAN-T5-XXL).We further discuss the challenges of jailbreak detection in terms of their effectiveness against known attacks.For further analysis, we release a dataset of model outputs across 3700 jailbreak prompts over 4 tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The hypothesis that episodic memories could lead to unpredictable and undesirable behavior has a strong parallel in existing work on the effects of examples given to large language models.
“Our hypothesis that episodic memories could be recalled and used in ways which lead to unpredictable and potentially undesirable agent behavior has a strong parallel in existing work on the effects of examples given to large language models.”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [98]
Few-shot in-context learning presents opportunities to undermine or defeat elements of large language models' training meant to keep them aligned to particular values such as being harmless.
“Several research groups have demonstrated that such few-shot in context learning presents opportunities to undermine or defeat elements of the models' training which are meant to keep them aligned to particular values such as being harmless [98], [99].”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- Eugene Bagdasaryan and Vitaly Shmatikov. 2022. Spinning language models: Risks of propaganda-as-a-service and countermeasures. · DOI
- Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Lev- eling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Spe- cia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine transla- tion. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Com- putational Linguistics. · DOI
- Ali Borji. 2023. A categorical archive of chatgpt failures. arXiv preprint arXiv:2302.03494. · arXiv
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. 9 https://github.com/AetherPrior/TrickLLM 10 Microsoft AFMR: https://bitly.ws/3eawI · DOI · Paper page
- Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023. Explore, establish, exploit: Red teaming language models from scratch. ArXiv, abs/2306.09442. · arXiv
- G. Cybenko, A. Giani, and P. Thompson. 2002. Cognitive hacking: a battle for the mind. Com- puter, 35(8):50-56. · DOI
- Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Jailbreaker: Automated jailbreak across multiple large language model chatbots. ArXiv, abs/2307.08715. · arXiv
- Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent ha- tred: A benchmark for understanding implicit hate speech. CoRR, abs/2109.05322. · arXiv
- Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022b. Language models as zero-shot planners: Extracting actionable knowl- edge for embodied agents. In ICML 2022, vol- ume 162 of Proceedings of Machine Learning Research, pages 9118-9147. PMLR.
- Yujin Huang, Terry Yue Zhuo, Qiongkai Xu, Han Hu, Xingliang Yuan, and Chunyang Chen. 2023. Training-free lexical backdoor attacks on lan- guage models. arXiv preprint arXiv:2302.04116. · arXiv
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reason- ers. · DOI · Paper page
- Judy Lambert and Mark Stevens. 2023. Chatgpt and generative ai technology: A mixed bag of concerns and new opportunities. Computers in the Schools, pages 1-25.
- Shaofeng Li, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Suguo Du, and Haojin Zhu. 2022. Backdoors against natural language processing: A review. IEEE Security & Privacy, 20(5):50-59. · DOI
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A sys- tematic survey of prompting methods in natu- ral language processing. ACM Comput. Surv., 55(9). · DOI · Paper page
- Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D. Griffin. 2023. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities. ArXiv, abs/2308.12833. · arXiv
- Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence rnns for text sum- marization. CoRR, abs/1602.06023. · arXiv
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. · DOI · Paper page
- Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527. · arXiv
- Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan. 2023. Latent jailbreak: A benchmark for evaluating text safety and out- put robustness of large language models. ArXiv, abs/2307.08487. · arXiv
- Xinyu Shen, Zeyuan Johnson Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. ArXiv, abs/2308.03825. · arXiv
- Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 139-150, Online. Association for Computational Linguis- tics. · DOI · Paper page
- Alexander Wei, Nika Haghtalab, and Jacob Stein- hardt. 2023. Jailbroken: How does llm safety training fail? ArXiv, abs/2307.02483. · arXiv
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652. · arXiv
- Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geof- frey Irving, and Iason Gabriel. 2022. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, page 214-229, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
- BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexan- dra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, and Albert Villanova del Moral et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. · DOI · Paper page
- Dongyu Yao, Jianshu Zhang, Ian G. Harris, and Marcel Carlsson. 2023. Fuzzllm: A novel and uni- versal fuzzing framework for proactively discov- ering jailbreak vulnerabilities in large language models. ArXiv, abs/2309.05274. · arXiv
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. · DOI · Paper page
- Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023a. Universal and transferable adversarial attacks on aligned language models. ArXiv, abs/2307.15043. · arXiv
Cited by
Sorted by citations
State of What Art? A Call for Multi-Prompt LLM Evaluation
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00681 · Cited by 113 · see citing works
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51 · see citing works
- Authors
- Rao, A., Vashistha, S., Naik, A., Aditya, S., & Choudhury, M.
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2024-05
- Pages
- 16802-16830
- ISSN
- 2522-2686
- OpenAlex
- W7131890740
- Cited by
- 19
- References
- 0
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 9 · 2023: 1
- FWCI
- 5.32
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- jailbreaking, large language models, jailbreak detection, prompt engineering, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
