CoChat
Sign Up
PreprintCited by 183Adversarial Robustness in Machine LearningSecurity and Verification in ComputingCryptographic Implementations and Security

Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

Open access
Mehrotra, Anay, Zampetakis, Manolis, Kassianik, Paul, Nelson, Blaine, Anderson, Hyrum, Singer, Yaron, Karbasi, Amin·2023·arXiv
DOI 10.48550/arxiv.2312.02119arXiv 2312.02119
Open this paper in CoChat — free accountOpen PDF

Abstract

While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM. TAP utilizes an attacker LLM to iteratively refine candidate (attack) prompts until one of the refined prompts jailbreaks the target. In addition, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks, reducing the number of queries sent to the target LLM. In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4-Turbo and GPT4o) for more than 80% of the prompts. This significantly improves upon the previous state-of-the-art black-box methods for generating jailbreaks while using a smaller number of queries than them. Furthermore, TAP is also capable of jailbreaking LLMs protected by state-of-the-art guardrails, e.g., LlamaGuard.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.02119

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing approaches often rely heavily on manual effort, such as designing seed prompts or heuristically crafting attack scenarios, limiting scalability.

    “Existing approaches (Chao et al., 2023;Mehrotra et al., 2023;Zhan et al., 2024) often rely heavily on manual effort, such as designing seed prompts (Wu et al., 2024b) or heuristically crafting attack scenarios (Wu et al., 2024c), limiting scalability.”

    From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Mehrotra et al., 2023

    The “Manually Discovered Jailbreaks” section says researchers have devoted significant effort to discovering jailbreaks manually, and “Automated and Black-Box Attacks” says one earlier method requires successful jailbreaks as seeds. But Mehrotra et al. describe their own method as automated and seed-free; these passages do not establish that existing approaches generally rely heavily on manual effort or that such reliance limits scalability. The specific seed-prompt and attack-scenario examples are cited to other works in the citing sentence. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Tree of Attacks with Pruning (TAP) involves iteratively expanding a tree wherein each node contains an adversarial injection attempt and associated metadata.

    “upon the recent Tree of Attacks with Pruning (TAP) jailbreak (Mehrotra et al., 2023).TAP involves iteratively expanding a tree wherein each node contains an adversarial injection attempt and some associated metadata.”

    From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Mehrotra et al., 2023

  • Evaluating a jailbreaking attack is subjective to the point of often requiring human or LLM judges, whereas product ranking order is precise and quantitative.

    “Evaluating a jailbreaking attack is subjective to the point of often requiring human (Zhu et al., 2023) or LLM (Mehrotra et al., 2023) judges, whereas product ranking order is precise and quantitative.”

    From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Mehrotra et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 64

  1. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks". In: (2024). arXiv: 2404.02151 [cs.CR]. URL: https://doi.org/10.48550/arXiv.2404.02151 (cit. on p. 6). · DOI
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Jun- whan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Fein- berg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 Technical Report. 2023. arXiv: 2305.10403 [cs.CL] (cit. on p. 7). · arXiv
  3. Abubakar Abid, Maheen Farooqi, and James Zou. "Persistent Anti-Muslim Bias in Large Language Models". In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. AIES '21. Virtual Event, USA: Association for Computing Machinery, 2021, pp. 298-306. ISBN: 9781450384735. DOI: 10.1145/3461702.3462624. URL: https://doi.org/10.1145/3461702.3462624 (cit. on pp. 3, 7). · DOI · Paper page
  4. Gabriel Alon and Michael Kamfonas. Detecting Language Model Attacks with Perplexity. 2023. arXiv: 2308.14132 [cs.CL]. URL: https://arxiv.org/abs/2308.14132 (cit. on pp. 3, 6). · arXiv
  5. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. 2016. arXiv: 1606.06565 [cs.AI] (cit. on p. 3). · arXiv
  6. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" In: Pro- ceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. FAccT '21. Virtual Event, Canada: Association for Computing Machinery, 2021, pp. 610-623. ISBN: 9781450383097. DOI: 10.1145/3442188.3445922. URL: https: //doi.org/10.1145/3442188.3445922 (cit. on pp. 3, 7). · DOI · Paper page
  7. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Pe- ter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Car- los Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models. 2022. arXiv: 2108.07258 [cs.LG] (cit. on pp. 3, 7). · arXiv
  8. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das- Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. 2022. arXiv: 2204.05862 [cs.CL] (cit. on p. 6). · arXiv
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. "Lan- guage Models are Few-Shot Learners". In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33.
  10. Curran Associates, Inc., 2020, pp. 1877-1901. URL: https://proceedings.neurips. cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf (cit. on p. 3).
  11. Jailbreak Chat. Jailbreak Chat. 2023. URL: https://jailbreakchat.com (cit. on pp. 3, 7).
  12. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. "Deep Reinforcement Learning from Human Preferences". In: Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/ file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf (cit. on p. 3).
  13. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. 2023. arXiv: 2306.09442 [cs.CL] (cit. on p. 6). · arXiv
  14. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. 2023. arXiv: 2310.08419 [cs.LG] (cit. on pp. 4-7, 9, 11, 12, 14, 25, 26, 36, 37, 39, 40, 42). · arXiv
  15. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. "Extracting Training Data from Large Language Models". In: 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, Aug. 2021, pp. 2633-2650. ISBN: 978-1-939133-24-3. URL: https://www.usenix.org/ conference/usenixsecurity21/presentation/carlini-extracting (cit. on p. 3).
  16. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. "MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots". In: Proceedings 2024 Network and Distributed System Security Symposium. NDSS 2024. Internet Society, 2024. DOI: 10.14722/ndss.2024. 24188. URL: http://dx.doi.org/10.14722/ndss.2024.24188 (cit. on p. 6). · DOI
  17. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. "Toxicity in ChatGPT: Analyzing Persona-Assigned Language Models". In: Findings of the Association for Computational Linguistics: EMNLP 2023.
  18. Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 1236-1270. DOI: 10 . 18653 / v1 / 2023 . findings-emnlp.88. URL: https://aclanthology.org/2023.findings-emnlp.88 (cit. on p. 3). · DOI · Paper page
  19. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. "HotFlip: White-Box Adversarial Examples for Text Classification". In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Ed. by Iryna Gurevych and Yusuke Miyao. Melbourne, Australia: Association for Com- putational Linguistics, July 2018, pp. 31-36. DOI: 10 . 18653 / v1 / P18 -2006. URL: https://aclanthology.org/P18-2006 (cit. on p. 42). · DOI
  20. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. 2022. arXiv: 2209.07858 [cs.CL] (cit. on pp. 3, 6). · arXiv
  21. Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. "Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers". In: 2018 IEEE Security and Privacy Workshops (SPW). 2018, pp. 50-56. DOI: 10.1109/SPW.2018. 00016 (cit. on p. 42). · DOI · Paper page
  22. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. "Gradient- based Adversarial Attacks against Text Transformers". In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Ed. by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5747-5757. DOI: 10.18653/v1/2021.emnlp-main.464. URL: https://aclanthology. org/2021.emnlp-main.464 (cit. on p. 42). · DOI
  23. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. "Explaining and Harness- ing Adversarial Examples". In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL: http://arxiv.org/abs/1412.6572 (cit. on p. 42). · arXiv
  24. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. "Aligning AI With Shared Human Values". In: International Conference on Learning Representations. 2021. URL: https://openreview.net/forum? id=dNy_RKzJacY (cit. on p. 3).
  25. Babak Hemmatian and Lav R. Varshney. Debiased Large Language Models Still Associate Muslims with Uniquely Violent Acts. 2022. arXiv: 2208.04417 [cs.CL] (cit. on pp. 3, 7). · arXiv
  26. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. 2023. arXiv: 2312.06674 [cs.CL] (cit. on pp. 4, 11, 38, 40). · arXiv
  27. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. "Automatically Auditing Large Language Models via Discrete Optimization". In: Proceedings of the 40th International Conference on Machine Learning. Ed. by Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett. Vol. 202. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 15307- 15329. URL: https://proceedings.mlr.press/v202/jones23a.html (cit. on pp. 6, 42).
  28. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. "BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset". In: Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track. 2023. URL: https://openreview.net/forum?id=g0QovXbFw3 (cit. on p. 3).
  29. Hadas Kotek, Rikker Dockum, and David Sun. "Gender Bias and Stereotypes in Large Language Models". In: Proceedings of The ACM Collective Intelligence Conference. CI '23. Delft, Netherlands: Association for Computing Machinery, 2023, pp. 12- 24. DOI: 10.1145/3582269.3615599. URL: https://doi.org/10.1145/3582269. 3615599 (cit. on p. 14). · DOI · Paper page
  30. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. "Exploiting Programmatic Behavior of LLMs: Dual-Use Through Stan- dard Security Attacks". In: 2024 IEEE Security and Privacy Workshops (SPW). 2024, pp. 132-143. DOI: 10.1109/SPW63631.2024.00018 (cit. on p. 6). · DOI · Paper page
  31. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. 2023. arXiv: 2305.13860 [cs.SE] (cit. on p. 3). · arXiv
  32. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. "Multi-step Jailbreaking Privacy Attacks on ChatGPT". In: The 2023 Conference on Empirical Methods in Natural Language Processing. 2023. URL: https://openreview. net/forum?id=ls4Pfsl2jZ (cit. on pp. 3, 7, 14).
  33. Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. "TextBugger: Generating Adversarial Text Against Real-world Applications". In: NDSS. The Internet Society, 2019 (cit. on p. 42).
  34. Raz Lapid, Ron Langberg, and Moshe Sipper. Open Sesame! Universal Black Box Jailbreaking of Large Language Models. 2023. arXiv: 2309.01446 [cs.CL] (cit. on pp. 3, 6). · arXiv
  35. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. "BERT- ATTACK: Adversarial Attack Against BERT Using BERT". In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Ed.
  36. by Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu. Online: Association for Computational Linguistics, Nov. 2020, pp. 6193-6202. DOI: 10 . 18653 / v1 / 2020 . emnlp-main.500. URL: https://aclanthology.org/2020.emnlp-main.500 (cit. on p. 42). · DOI
  37. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebas- tian Riedel, and Douwe Kiela. "Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks". In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 9459-9474. URL: https : // proceedings. neurips . cc / paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf (cit. on p. 3).
  38. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. "AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models". In: The Twelfth International Conference on Learning Representations. 2024. URL: https://openreview. net/forum?id=7Jwpw4qKkb (cit. on pp. 6, 42).
  39. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, El- ham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal". In: Forty-first International Conference on Machine Learning. 2024. URL: https://openreview.net/forum?id=f3TUipYU3U (cit. on p. 6).
  40. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. "Combating Adversar- ial Misspellings with Robust Word Recognition". In: Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics. Ed. by Anna Korhonen, David Traum, and Lluís Màrquez. Florence, Italy: Association for Computational Linguistics, July 2019, pp. 5582-5591. DOI: 10.18653/v1/P19-1561. URL: https: //aclanthology.org/P19-1561 (cit. on p. 42). · DOI
  41. Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking Large Language Models via Adversarial In-Context Learning. 2023. arXiv: 2311.09948 [cs.LG] (cit. on p. 6). · arXiv
  42. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Any- thing Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Lan- guage Models. 2024. arXiv: 2308.03825 [cs.CR] (cit. on p. 6). · arXiv
  43. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. "The Woman Worked as a Babysitter: On Biases in Language Generation". In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  44. Ed. by Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3407-3412. DOI: 10.18653/ v1/D19-1339. URL: https://aclanthology.org/D19-1339 (cit. on pp. 3, 7). · DOI · Paper page
  45. Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. 2023. arXiv: 2311.03348 [cs.CL] (cit. on p. 6). · arXiv
  46. Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Joseph Konan, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Desh- mukh, Michael Kuhlmann, Bhiksha Raj, and Rita Singh. LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model. 2023. arXiv: 2310.04445 [cs.CL] (cit. on p. 6). · arXiv
  47. Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. "Universal Ad- versarial Attacks with Natural Triggers for Text Classification". In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Ed. by Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou. Online: Association for Computational Linguistics, June 2021, pp. 3724-3733. DOI: 10.18653/v1/2021.naacl-main.291. URL: https://aclanthology.org/2021.naacl-main.291 (cit. on p. 42). · DOI
  48. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. "Intriguing Properties of Neural Networks". In: ICLR (Poster). 2014 (cit. on p. 42).
  49. Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. "It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations". In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault. Online: Association for Computational Linguistics, July 2020, pp. 2920-2935. DOI: 10.18653/v1/2020.acl- main.263. URL: https://aclanthology.org/2020.acl-main.263 (cit. on p. 42). · DOI
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xi- ang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. arXiv: 2307.09288 [cs.CL] (cit. on p. 6). · arXiv
  51. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. "Universal Adversarial Triggers for Attacking and Analyzing NLP". In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  52. Ed. by Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 2153-2162. DOI: 10.18653/v1/D19-1221. URL: https://aclanthology.org/D19-1221 (cit. on pp. 6, 42). · DOI · Paper page
  53. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. "Jailbroken: How Does LLM Safety Training Fail?" In: Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL: https://openreview.net/forum?id=jA235JGM09 (cit. on pp. 3, 6, 7, 10, 14, 38).
  54. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery". In: Advances in Neural Information Processing Sys- tems. Ed. by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. Vol. 36. Curran Associates, Inc., 2023, pp. 51008-51025. URL: https://proceedings. neurips.cc/paper_files/paper/2023/file/a00548031e4647b13042c97c922fadf1- Paper-Conference.pdf (cit. on p. 42).
  55. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter brian, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 24824-24837. URL: https://proceedings. neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4- Paper-Conference.pdf (cit. on p. 4).
  56. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey. 2023. arXiv: 2307.12966 [cs.CL] (cit. on p. 3). · arXiv
  57. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for Safety in Open-domain Chatbots. 2021. arXiv: 2010.07079 [cs.CL] (cit. on p. 3). · arXiv
  58. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. "A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models". In: Findings of the Association for Computational Linguistics ACL 2024. Ed. by Lun-Wei Ku, Andre Martins, and Vivek Srikumar. Bangkok, Thailand and virtual meeting: Association for Computational Linguistics, Aug. 2024, pp. 7432-7449. DOI: 10.18653/v1/2024. findings-acl.443. URL: https://aclanthology.org/2024.findings-acl.443 (cit. on p. 6). · DOI
  59. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. 2023. arXiv: 2309.10253 [cs.AI] (cit. on pp. 3, 6). · arXiv
  60. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models". In: Advances in Neural Information Processing Systems. Ed. by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. Vol. 36. Curran Associates, Inc., 2023, pp. 11809-11822. URL: https://proceedings.neurips.cc/ paper _ files / paper / 2023 / file / 271db9922b8d1f4dd7aaef84ed5ac703 -Paper - Conference.pdf (cit. on p. 7).
  61. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. 2024. arXiv: 2401.06373 [cs.CL] (cit. on p. 6). · arXiv
  62. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. "Word-level Textual Adversarial Attacking as Combinatorial Optimization". In: Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics. Ed. by Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault. Online: Association for Computational Linguistics, July 2020, pp. 6066-6080. DOI: 10.18653/v1/2020.acl-main.540. URL: https://aclanthology.org/2020.acl- main.540 (cit. on p. 42). · DOI
  63. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. 2020. arXiv: 1909.08593 [cs.CL] (cit. on p. 3). · arXiv
  64. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. 2023. arXiv: 2307.15043 [cs.CL] (cit. on pp. 3, 6, 7, 9, 11, 12). · arXiv

Cited by

Sorted by citations

Authors
Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., & Karbasi, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
61065-61105
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2312.02119
arXiv
2312.02119
OpenAlex
W4415797457
Cited by
183
References
0
Citations by year
2026: 94 · 2025: 66 · 2024: 23
FWCI
44.23
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Cryptographic Implementations and Security
Keywords
jailbreaking, large language models, black-box attacks, language model security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review