CoChat
Sign Up
PreprintCited by 861Adversarial Robustness in Machine LearningHate Speech and Cyberbullying DetectionTopic Modeling

The DOI matches its DataCite record.

"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

Open access
Shen, Xinyue, Chen, Zeyuan, Backes, Michael, Shen, Yun, Zhang, Yang·2023·arXiv
DOI 10.48550/arxiv.2308.03825arXiv 2308.03825
Open this paper in CoChat — free accountOpen PDF

Abstract

The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.03825

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Red teamers employed diverse attack strategies including obfuscation methods, prompt injection, template-based attacks, and decomposition strategies.

    “Red teamers also experimented with various obfuscation methods, such as encoding schemes and text-in-image attacks.”

    From Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (2025) · cites this paper as Shen et al., 2023

    The Trending Communities section describes diverse jailbreak strategies, including prompt injection. The Results section discusses obfuscation as an attack used in prior work, and §5.4 tests paraphrasing and typos. Our copy does not substantiate the citing sentence’s specific examples of encoding schemes or text-in-image attacks, or the broader claim’s template-based and decomposition strategies. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 63

  1. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin De- hghani, Juan Diego Bermeo, Maria Korobeynikova, and Fab- rizio Gilardi. Open-Source Large Language Models Out- perform Crowd Workers and Approach ChatGPT in Text- Annotation Tasks. CoRR abs/2307.02179, 2023. 17 · arXiv
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Ke- fan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernán- dez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Gar- cia, Sebastian Gehrmann, Lucas Gonzalez, and et al. PaLM 2 Technical Report. CoRR abs/2305.10403, 2023. 17 · arXiv
  3. Eugene Bagdasaryan and Vitaly Shmatikov. Spinning Lan- guage Models: Risks of Propaganda-As-A-Service and Coun- termeasures. In IEEE Symposium on Security and Privacy (S&P), pages 769-786. IEEE, 2022. 13
  4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A Multitask, Multilingual, Multimodal Evaluation of Chat- GPT on Reasoning, Hallucination, and Interactivity. CoRR abs/2302.04023, 2023. 3 · arXiv
  5. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The Pushshift Reddit Dataset. In International Conference on Web and Social Me- dia (ICWSM), pages 830-839. AAAI, 2020. 4
  6. Marzieh Bitaab, Haehyun Cho, Adam Oest, Zhuoer Lyu, Wei Wang, Jorij Abraham, Ruoyu Wang, Tiffany Bao, Yan Shoshi- taishvili, and Adam Doupé. Beyond Phish: Toward Detecting Fraudulent e-Commerce Websites at Scale. In IEEE Sympo- sium on Security and Privacy (S&P), pages 2566-2583. IEEE, 2023. 3
  7. Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nico- las Papernot. Bad Characters: Imperceptible NLP Attacks. In IEEE Symposium on Security and Privacy (S&P), pages 1987-2004. IEEE, 2022. 13
  8. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Lan- guage Models. In USENIX Security Symposium (USENIX Se- curity), pages 2633-2650. USENIX, 2021. 13
  9. Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. BadNL: Back- door Attacks Against NLP Models with Semantic-preserving Improvements. In Annual Computer Security Applications Conference (ACSAC), pages 554-569. ACSAC, 2021. 13
  10. Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. PLUE: Language Understanding Evaluation Bench- mark for Privacy Policies in English. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 352-365. ACL, 2023. 3
  11. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Annual Conference on Neural Information Processing Systems (NIPS), pages 4299- 4307. NIPS, 2017. 17
  12. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jail- breaker: Automated Jailbreak Across Multiple Large Lan- guage Model Chatbots. CoRR abs/2307.08715, 2023. 12 · arXiv
  13. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In An- nual Meeting of the Association for Computational Linguistics (ACL), pages 320-335. ACL, 2022. 17
  14. Yunhe Feng, Pradhyumna Poralla, Swagatika Dash, Kaicheng Li, Vrushabh Desai, and Meikang Qiu. The Impact of Chat- GPT on Streaming Media: A Crowdsourced and Data-Driven Analysis using Twitter and Reddit. In IEEE International Conference on Big Data Security on Cloud, High Performance and Smart Computing and Intelligent Data and Security (Big- DataSecurity/HPSC/IDS), pages 222-227. IEEE, 2023. 4
  15. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you've asked for: A Comprehensive Analysis of Novel Prompt Injec- tion Threats to Application-Integrated Large Language Mod- els. CoRR abs/2302.12173, 2023. 13 · arXiv
  16. Julian Hazell. Large Language Models Can Be Used To Effectively Scale Spear Phishing Campaigns. CoRR abs/2305.06972, 2023. 1, 3, 13 · arXiv
  17. Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. MGTBench: Benchmarking Machine-Generated Text Detection. CoRR abs/2303.14822, 2023. 13 · arXiv
  18. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettle- moyer. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL- HLT), pages 1875-1885. ACL, 2018. 13
  19. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is ChatGPT A Good Translator? A Prelim- inary Study. CoRR abs/2301.08745, 2023. 3 · arXiv
  20. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT Really Robust? A Strong Baseline for Natural Lan- guage Attack on Text Classification and Entailment. In AAAI Conference on Artificial Intelligence (AAAI), pages 8018- 8025. AAAI, 2020. 13
  21. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security At- tacks. CoRR abs/2302.05733, 2023. 1, 2, 3, 13 · arXiv
  22. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the Reli- ability of Watermarks for Large Language Models. CoRR abs/2306.04634, 2023. 11 · arXiv
  23. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step Jailbreaking Privacy Attacks on ChatGPT. CoRR abs/2304.05197, 2023. 12, 13 · arXiv
  24. Xuezixiang Li, Yu Qu, and Heng Yin. PalmTree: Learning an Assembly Language Model for Instruction Embedding. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 3236-3251. ACM, 2021. 3
  25. Zilong Lin, Jian Cui, Xiaojing Liao, and XiaoFeng Wang. Malla: Demystifying Real-world Large Language Model In- tegrated Malicious Services. CoRR abs/2401.03315, 2024. 2, 13 · arXiv
  26. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hi- roaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Nat- ural Language Processing. ACM Computing Surveys, 2023. 3
  27. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jail- breaking ChatGPT via Prompt Engineering: An Empirical Study. CoRR abs/2305.13860, 2023. 12 · arXiv
  28. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella Béguelin. Analyzing Leak- age of Personally Identifiable Information in Language Mod- els. In IEEE Symposium on Security and Privacy (S&P), pages 346-363. IEEE, 2023. 13
  29. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloun- dou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A Holistic Approach to Undesired Content Detection in the Real World. CoRR abs/208.03274, 2022. 1, 2, 12
  30. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform Manifold Approximation and Projection. The Journal of Open Source Software, 2018. 5
  31. Pasquale De Meo, Emilio Ferrara, Giacomo Fiumara, and Alessandro Provetti. Generalized Louvain method for com- munity detection in large networks. In International Confer- ence on Intelligent Systems Design and Applications (ISDA), pages 88-93. IEEE, 2011. 6, 17
  32. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethink- ing the Role of Demonstrations: What Makes In-Context Learning Work? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11048-11064. ACL, 2022. 18
  33. Jaron Mink, Licheng Luo, Natã M. Barbosa, Olivia Figueira, Yang Wang, and Gang Wang. DeepPhish: Understanding User Trust Towards Artificially Generated Profiles in Online Social Networks. In USENIX Security Symposium (USENIX Security), pages 1669-1686. USENIX, 2022. 13
  34. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. Quantifying Pri- vacy Risks of Masked Language Models Using Membership Inference Attacks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8332-8347. ACL, 2022. 13
  35. Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristofaro. "And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM), pages 452-463. AAAI, 2020. 3
  36. Mark E. J. Newman. Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 2006. 17
  37. OpenAI. GPT-4 Technical Report. CoRR abs/2303.08774, 2023. 1, 3, 8, 17 · arXiv
  38. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Annual Conference on Neural In- formation Processing Systems (NeurIPS). NeurIPS, 2022. 1, 3
  39. Alessandro Pegoraro, Kavita Kumari, Hossein Fereidooni, and Ahmad-Reza Sadeghi. To ChatGPT, or not to ChatGPT: That is the question! CoRR abs/2304.01487, 2023. 3 · arXiv
  40. Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can Large Language Models Reason about Program Invariants? In International Conference on Machine Learning (ICML). JMLR, 2023. 3
  41. Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. CoRR abs/2211.09527, 2022. 13 · arXiv
  42. Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe Diffusion: On the Gen- eration of Unsafe Images and Hateful Memes From Text-To- Image Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2023. 1, 3, 13
  43. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 3980-3990. ACL, 2019. 5
  44. Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Annual Meeting of the Associa- tion for Computational Linguistics (ACL), pages 4902-4912. ACL, 2020. 11
  45. Omar Shaikh, Hongxin Zhang, William Held, Michael Bern- stein, and Diyi Yang. On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 4454-4470. ACL, 2023. 8
  46. Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. CoRR abs/2304.08979, 2023. 13 · arXiv
  47. Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In ACM SIGSAC Con- ference on Computer and Communications Security (CCS), pages 2659-2673. ACM, 2022. 9
  48. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feed- back. CoRR abs/2009.01325, 2020. 17 · arXiv
  49. Jörg Tiedemann and Santhosh Thottingal. OPUS-MT -Build- ing open translation services for the World. In Conference of the European Association for Machine Translation (EAMT), pages 479-480. European Association for Machine Transla- tion, 2020. 11
  50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roz- ière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. LLaMA: Open and Efficient Foundation Language Mod- els. CoRR abs/2302.13971, 2023. 3 · arXiv
  51. Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022. 13
  52. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Confer- ence on Neural Information Processing Systems (NIPS), pages 5998-6008. NIPS, 2017. 3
  53. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. DecodingTrust: A Compre- hensive Assessment of Trustworthiness in GPT Models. CoRR abs/2306.11698, 2023. 13 · arXiv
  54. Zijie J. Wang, Fred Hohman, and Duen Horng Chau. Wiz- Map: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 516- 523. ACL, 2023. 5
  55. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jail- broken: How Does LLM Safety Training Fail? CoRR abs/2307.02483, 2023. 8, 12 · arXiv
  56. Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, and Bo Li. Detecting AI Trojans Using Meta Neural Analysis. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2021. 13
  57. Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yev- geniy Vorobeychik, and Chaowei Xiao. CodeIPPrompt: Intel- lectual Property Infringement Assessment of Code Language Models. In International Conference on Machine Learning (ICML). JMLR, 2023. 13
  58. Savvas Zannettou, Joel Finkelstein, Barry Bradlyn, and Jeremy Blackburn. A Quantitative Approach to Understand- ing Online Antisemitism. In International Conference on Web and Social Media (ICWSM), pages 786-797. AAAI, 2020. 6
  59. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: An Open Bilingual Pre-trained Model. In International Conference on Learning Represen- tations (ICLR), 2023. 3, 8, 17
  60. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jail- break Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. CoRR abs/2401.06373, 2024. 2 · arXiv
  61. Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G. Parker, and Munmun De Choudhury. Synthetic Lies: Understand- ing AI-Generated Misinformation and Evaluating Algorith- mic and Human Solutions. In Annual ACM Conference on Human Factors in Computing Systems (CHI), pages 436:1- 436:20. ACM, 2023. 1, 3, 13
  62. Yiming Zhu, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks. CoRR abs/2304.10145, 2023. 17 · arXiv
  63. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043, 2023. 12, 13 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y.
Affiliations
Helmholtz Center for Information Security (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
1671-1685
ISSN
1543-7221 (ISSN-L)
DOI
10.48550/arxiv.2308.03825
arXiv
2308.03825
OpenAlex
W4405181600
Cited by
861
References
0
Citations by year
2026: 310 · 2025: 290 · 2024: 230 · 2023: 31
FWCI
131.03
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Figshare (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
  • CISPA Helmholtz Center (cc-by)
Funders
  • Universitas Brawijaya (101057917)
  • European Health and Digital Executive Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
jailbreak prompts, large language models, prompt injection, privilege escalation, LLM misuse, ChatGPT, GPT-4, prompt optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review