CoChat
Sign Up
PreprintCited by 4Adversarial Robustness in Machine LearningSecurity and Verification in ComputingTopic Modeling

Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

Open access
Schulhoff, Sander, Pinto, Jeremy, Khan, Anaum, Bouchard, Louis-François, Si, Chenglei, Anati, Svetlina, Tagliabue, Valen, Kost, Anson Liu +2 more·2023·arXiv
DOI 10.48550/arxiv.2311.16119arXiv 2311.16119
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.16119

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Jailbreaks aim to escape the safety behavior trained into an LLM and often do not specifically conflict with a model's previous instructions.

    “Jailbreaks Another well-known attack vector is jailbreak, which specifically aims to escape the safety behavior that is trained into an LLM.As such, they often don't specifically conflict with a model's previous instructions.”

    From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Schulhoff et al., 2023

    Section 5.1 distinguishes prompt injection, which overrides original instructions, from jailbreaking, which elicits unintended behavior. Appendix D.12.6 describes a jailbreak that exploits a conflict with a model’s training to ignore malicious instructions. The paper does not establish that jailbreaks specifically target safety-trained behavior in general or that they often avoid conflicting with previous instructions; its definition of jailbreaking is broader. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • System message extraction attacks are straightforward to execute even when models are told to avoid repeating their prompt.

    “These attacks are straightforward to execute (Perez & Ribeiro, 2022;Zhang & Ippolito, 2023;Schulhoff et al., 2023)-even when models are told to avoid repeating their prompt-by using attacks such as:”

    From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Schulhoff et al., 2023

    Section 4.4 (page unavailable) describes a simple attack that made a model reveal a hidden key despite an instruction not to expose it. Appendix H, Level 2 (page unavailable), identifies this as a prompt-leak attack. Those passages support the ease of extracting a protected item, but not the broader claim that extracting an entire system message is straightforward when the model is told not to repeat its prompt. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.

    “Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.”

    From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Schulhoff et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 50

  1. Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. 2023. (Ab)using Images and Sounds for Indirect Instruction Injection in Multi- Modal LLMs. ArXiv, abs/2307.10490. · arXiv
  2. Eugene Bagdasaryan and Vitaly Shmatikov. 2023. Ceci n'est pas une pomme: Adversarial Illusions in Multi- Modal Embeddings. ArXiv, abs/2308.11804. · arXiv
  3. Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas- tian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for read- ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678. · DOI
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Etha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the Opportunities and Risks of Foundation Mod- els. ArXiv, abs/2108.07258. · arXiv
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in neural information process- ing systems.
  6. Nicholas Carlini, Milad Nasr, Christopher A Choquette- Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Flo- rian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? ArXiv, abs/2306.15447. · arXiv
  7. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raf- fel. 2020. Extracting Training Data from Large Lan- guage Models. In USENIX Security Symposium.
  8. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is ChatGPT's behavior changing over time? ArXiv, abs/2307.09009. · arXiv
  9. Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K Gupta, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Misusing Tools in Large Language Models With Visual Adversarial Examples . ArXiv, abs/2310.03185. · arXiv
  10. Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nel- son Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan, Danny Hernandez, Tris- tan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran- Johnson, Dario Amodei, Tom B. Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. ArXiv, abs/2209.07858. · arXiv
  11. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. PAL: Program-aided Language Models. In International Conference on Machine Learning.
  12. Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. Communications of the ACM, 64:86 - 92.
  13. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020.
  14. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You've Signed Up For: Compromis- ing Real-World LLM-Integrated Applications with Indirect Prompt Injection. ArXiv, abs/2302.12173. · arXiv
  15. Zihan Guan, Zihao Wu, Zhengliang Liu, Dufan Wu, Hui Ren, Quanzheng Li, Xiang Li, and Ninghao Liu. 2023. CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study. ArXiv, abs/2307.11346. · arXiv
  16. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Ex- ploiting programmatic behavior of LLMs: Dual- use through standard security attacks. ArXiv, abs/2302.05733. · arXiv
  17. Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Ha- jishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt wayward- ness: The curious case of discretized interpretation of continuous prompts. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. · DOI
  18. Alexey Kirichenko, Markus Christen, Florian Grunow, and Dominik Herrmann. 2020. Best practices and recommendations for cybersecurity service providers. The ethics of cybersecurity, pages 299-316. · DOI
  19. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science. · DOI · Paper page
  20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys.
  21. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023a. Prompt Injection at- tack against LLM-integrated Applications. ArXiv, abs/2306.05499. · arXiv
  22. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023b. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. ArXiv, abs/2305.13860. · arXiv
  23. Yi-Hsien Liu, Tianle Han, Siyuan Ma, Jia-Yu Zhang, Yuanyu Yang, Jiaming Tian, Haoyang He, Antong Li, Mengshen He, Zheng Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. 2023c. Summary of ChatGPT- Related Research and Perspective Towards the Future of Large Language Models. ArXiv, abs/2304.01852. · arXiv
  24. Robert L. Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models. In Find- ings of the Association for Computational Linguistics: ACL 2022.
  25. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. ArXiv, abs/2303.17651. · arXiv
  26. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, 2203.02155. · arXiv
  27. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. arXiv, 2211.09527. · arXiv
  28. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. 2023. Visual Adversarial Examples Jailbreak Large Language Models. ArXiv, 2306.13213. · arXiv
  29. Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. 2023. Tricking LLMs into disobedience: Understanding, analyzing, and preventing jailbreaks. ArXiv, 2305.14965. · arXiv
  30. Maria Rigaki and Sebastian Garcia. 2020. A Survey of Privacy Attacks in Machine Learning. ACM Comput- ing Surveys.
  31. Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B- Parameter Open-Access Multilingual Language Model. ArXiv, https://arxiv.org/abs/2211.05100. · arXiv
  32. Christian Schlarmann and Matthias Hein. 2023. On the Adversarial Robustness of Multi-Modal Foundation Models . In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision.
  33. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In Annual Meeting of the Association for Computational Linguistics.
  34. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023. Plug and Pray: Exploiting off-the-shelf components of Multi-Modal Models. ArXiv, abs/2307.14539. · arXiv
  35. Xinyu Shen, Zeyuan Johnson Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. "do anything now": Characterizing and evaluating in-the-wild jail- break prompts on large language models. ArXiv, abs/2308.03825. · arXiv
  36. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics. · DOI · Paper page
  37. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber, and Lijuan Wang. 2023. Prompting GPT-3 to be reliable. In ICLR.
  38. Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mah- moud Khalil, Nancy Fulda, and David Wingate. 2022. An information-theoretic approach to prompt engi- neering without ground truth labels. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). · DOI · Paper page
  39. M. A. van Wyk, M. Bekker, X. L. Richards, and K. J. Nixon. 2023. Protect Your Prompts: Protocols for IP Protection in LLM Applications. ArXiv.
  40. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George F. Foster. 2023. Prompt- ing PaLM for Translation: Assessing Strategies and Performance. In Annual Meeting of the Association for Computational Linguistics.
  41. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Ya- mada, and Jordan Boyd-Graber. 2019. Trick Me If You Can: Human-in-the-loop Generation of Adver- sarial Question Answering Examples. Transactions of the Association of Computational Linguistics.
  42. Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Conference of the North American Chapter of the Association for Computational Lin- guistics.
  43. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? In Conference on Neural Information Process- ing Systems.
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Conference on Neural Information Processing Sys- tems.
  45. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending ChatGPT against jailbreak attack via self-reminder. Physical Sciences -Article. · DOI
  46. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. Low-Resource Languages Jailbreak GPT-4. ArXiv, abs/2310.02446. · arXiv
  47. Shui Yu. 2013. Distributed Denial of Service Attack and Defense. Springer Publishing Company, Incor- porated.
  48. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can't prompt: How non-ai experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Con- ference on Human Factors in Computing Systems, CHI '23. Association for Computing Machinery. · DOI · Paper page
  49. Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, and Hai Jin. 2023. AdvCLIP: Downstream-agnostic Adversarial Examples in Mul- timodal Contrastive Learning . In ACM International Conference on Multimedia.
  50. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Weirong Ye, Neil Zhenqiang Gong, Yue Zhang, and Xingxu Xie. 2023. PromptBench: Towards Evaluating the Ro- bustness of Large Language Models on Adversarial Prompts. ArXiv, abs/2306.04528. · arXiv

Cited by

Sorted by citations

Authors
Schulhoff, S., Pinto, J., Khan, A., Bouchard, L.-F., Si, C., Anati, S., Tagliabue, V., Kost, A. L., Carnahan, C., & Boyd-Graber, J.
Affiliations
  • University of Maryland, College Park (US)
  • Technical University of Sofia (BG)
  • University of Milan (IT)
  • New York University (US)
  • University of Arizona (US)
ORCID
Boyd-Graber, Jordan
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.16119
arXiv
2311.16119
OpenAlex
W4389157312
Cited by
4
References
0
Citations by year
2025: 1 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
prompt injection, jailbreaking, large language models, adversarial prompts, LLM security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review