CoChat
Sign Up
PreprintCited by 4

MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Deng, Gelei, Liu, Yi, Li, Yuekang, Wang, Kailong, Zhang, Ying, Li, Zefeng, Wang, Haoyu, Zhang, Tianwei +1 more·2023·arXiv
DOI 10.48550/arxiv.2307.08715arXiv 2307.08715
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) services due to their exceptional proficiency in understanding and generating human-like text. LLM chatbots, in particular, have seen widespread adoption, transforming human-machine interactions. However, these LLM chatbots are susceptible to "jailbreak" attacks, where malicious users manipulate prompts to elicit inappropriate or sensitive responses, contravening service policies. Despite existing attempts to mitigate such threats, our research reveals a substantial gap in our understanding of these vulnerabilities, largely due to the undisclosed defensive measures implemented by LLM service providers. In this paper, we present Jailbreaker, a comprehensive framework that offers an in-depth understanding of jailbreak attacks and countermeasures. Our work makes a dual contribution. First, we propose an innovative methodology inspired by time-based SQL injection techniques to reverse-engineer the defensive strategies of prominent LLM chatbots, such as ChatGPT, Bard, and Bing Chat. This time-sensitive approach uncovers intricate details about these services' defenses, facilitating a proof-of-concept attack that successfully bypasses their mechanisms. Second, we introduce an automatic generation method for jailbreak prompts. Leveraging a fine-tuned LLM, we validate the potential of automated jailbreak generation across various commercial LLM chatbots. Our method achieves a promising average success rate of 21.58%, significantly outperforming the effectiveness of existing techniques. We have responsibly disclosed our findings to the concerned service providers, underscoring the urgent need for more robust defenses. Jailbreaker thus marks a significant step towards understanding and mitigating jailbreak threats in the realm of LLM chatbots.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.08715

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AdvAgent is specifically designed for black-box web agent attacks, incorporating model feedback for superior performance.

    “Unlike existing LLM red-teaming approaches (Deng et al., 2023; Ge et al., 2023; Paulus et al., 2024), AdvAgent is specifically designed for black-box web agent attacks, incorporating model feedback for superior performance.”

    From AdvAgent: Controllable Blackbox Red-teaming on Web Agents (2024) · cites this paper as Deng et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 37

  1. V. Atlidakis, P. Godefroid, and M. Polishchuk, "RESTler: Stateful REST API Fuzzing," 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 748-758, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:85465006
  2. E. Bagdasaryan and V. Shmatikov, "Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures," in S&P. IEEE, 2022, pp. 769-786.
  3. F. L. Bauer, Cæsar Cipher. Boston, MA: Springer US, 2011, pp. 180- 180. [Online]. Available: https://doi.org/10.1007/978-1-4419-5906-5 162 · DOI
  4. I. Beltagy, K. Lo, and A. Cohan, "Scibert: A pretrained language model for scientific text," in EMNLP, 2019.
  5. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" in FAccT, pp. 610-623.
  6. I. Cohen, Y. Huang, J. Chen, J. Benesty, J. Benesty, J. Chen, Y. Huang, and I. Cohen, "Pearson correlation coefficient," Noise reduction in speech processing, pp. 1-4, 2009.
  7. G. Deng, Z. Zhang, Y. Li, Y. Liu, T. Zhang, Y. Liu, G. Yu, and D. Wang, "NAUTILUS: Automated RESTful API Vulnerability Detection," in USENIX Security Symposium, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:260777640
  8. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, "Re- alToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models," in EMNLP, 2020, pp. 3356-3369.
  9. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," in arXiv preprint, 2023.
  10. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, "Large language models for software engi- neering: A systematic literature review," 2023.
  11. E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Kr- usche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, "Chatgpt for good? on opportunities and challenges of large language models for education," Learning and Individual Differences, vol. 103, p. 102274, 2023.
  12. D. Lawley, "A Generalization of Fisher's Z Test," Biometrika, vol. 30, no. 1/2, pp. 180-187, 1938.
  13. Y. Liu, "RESTInfer: Automated Inferring Parameter Constraints from Natural Language RESTful API Descriptions," in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1816-1818. [Online]. Available: https://doi.org/10.1145/3540250.3559078 · DOI
  14. Y. Liu, Y. Li, G. Deng, Y. Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, "Morest: Model-based RESTful API Testing with Execution Feedback," 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pp. 1406-1417, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248392251
  15. Y. Liu, Y. Li, Y. Liu, R. Wan, R. Wu, and Q. Liu, "Morest: Industry practice of automatic restful api testing," in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '22. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3551349.3559498 · DOI
  16. P. Manakul, A. Liusie, and M. J. Gales, "Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models," arXiv preprint, 2023.
  17. N. McKenna, T. Li, L. Cheng, M. J. Hosseini, M. Johnson, and M. Steedman, "Sources of Hallucination by Large Language Models on Inference Tasks," arXiv preprint, 2023.
  18. K. Mei, Z. Li, Z. Wang, Y. Zhang, and S. Ma, "NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models," in ACL, 2023.
  19. J. Oppenlaender, R. Linder, and J. Silvennoinen, "Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering," arXiv preprint, 2023.
  20. F. Perez and I. Ribeiro, "Ignore Previous Prompt: Attack Techniques For Language Models," in NeurIPS ML Safety Workshop, 2022.
  21. R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng, "Automatic Prompt Optimization with Gradient Descent and Beam Search," arXiv preprint, 2023.
  22. A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury, "Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks," arXiv preprint, 2023.
  23. L. Reynolds and K. McDonell, "Prompt programming for large language models: Beyond the few-shot paradigm," in CHI EA, 2021.
  24. A. Salem, M. Backes, and Y. Zhang, "Get a Model! Model Hijacking Attack Against Machine Learning Models," in NDSS, 2022.
  25. M. Shanahan, K. McDonell, and L. Reynolds, "Role-play with large language models," arXiv preprint, 2023.
  26. W. M. Si, M. Backes, J. Blackburn, E. D. Cristofaro, G. Stringhini, S. Zannettou, and Y. Zhang, "Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots," in CCS, 2022, pp. 2659- 2673.
  27. W. M. Si, M. Backes, Y. Zhang, and A. Salem, "Two-in-One: A Model Hijacking Attack Against Text Generation Models," arXiv preprint, 2023.
  28. W. Sun, Z. Shi, S. Gao, P. Ren, M. de Rijke, and Z. Ren, "Contrastive Learning Reduces Hallucination in Conversations," arXiv preprint, 2022.
  29. t. Wang, J. Zhang, G. Bai, R. Ko, and J. S. Dong, "It's Not Just the Site, It's the Contents: Intra-domain Fingerprinting Social Media Websites Through CDN Bursts," in 30th The Web Conference (WWW), 2021.
  30. T. Van Ede, H. Aghakhani, N. Spahn, R. Bortolameotti, M. Cova, A. Continella, M. van Steen, A. Peter, C. Kruegel, and G. Vigna, "Deepcase: Semi-supervised Contextual Analysis of Security Events," in IEEE S&P, 2022, pp. 522-539.
  31. E. Viglianisi, M. Dallago, and M. Ceccato, "RESTTESTGEN: Automated Black-Box Testing of RESTful APIs," 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 142-152, 2020. [Online]. Available: https: //api.semanticscholar.org/CorpusID:221107287
  32. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT," arXiv preprint, 2023.
  33. Y. Wolf, N. Wies, Y. Levine, and A. Shashua, "Fundamental limitations of alignment in large language models," arXiv preprint, 2023.
  34. J. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, "Why Johnny Can't Prompt: How Non-AI Experts Try (and fail) to Design LLM Prompts," in CHI, 2023, pp. 1-21.
  35. Y. Zhang, M. M. A. Kabir, Y. Xiao, D. Yao, and N. Meng, "Automatic detection of java cryptographic api misuses: Are we there yet?" IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 288-303, 2023.
  36. Z. Zhang, L. Lyu, X. Ma, C. Wang, and X. Sun, "Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models," in EMNLP, 2022, pp. 355-372.
  37. Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, "Large Language Models are Human-level Prompt Engineers," arXiv preprint, 2022.

Cited by

Sorted by citations

Authors
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2307.08715
arXiv
2307.08715
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review