CoChat
Sign Up
PreprintCited by 20Adversarial Robustness in Machine LearningHate Speech and Cyberbullying DetectionSecurity and Verification in Computing

The DOI matches its DataCite record.

Testing the Limits of Jailbreaking Defenses with the Purple Problem

Open access
Kim, Taeyoun, Kotha, Suhas, Raghunathan, Aditi·2024·arXiv
DOI 10.48550/arxiv.2403.14725arXiv 2403.14725
Open this paper in CoChat — free accountOpen PDF

Abstract

The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing undesirable responses. We critically examine the two stages of the defense pipeline: (i) defining what constitutes unsafe outputs, and (ii) enforcing the definition via methods such as input processing or fine-tuning. To test the efficacy of existing enforcement mechanisms, we consider a simple and well-specified definition of unsafe outputs--outputs that contain the word "purple". Surprisingly, existing fine-tuning and input defenses fail on this simple problem, casting doubt on whether enforcement algorithms can be robust for more complicated definitions. We find that real safety benchmarks similarly test enforcement for a fixed definition. We hope that future research can lead to effective/fast enforcement as well as high quality definitions used for enforcement and evaluation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.14725

Cites

Statistics

Figures

Figure 3 : Figure 3: Attack perplexity under Llama-IT.We take natural prompts, prompts with adversarial suffixes, and prompts with adaptively trained adversarial suffixes and measure their log perplexity. The perplexity defense can perfectly distinguish the basic attacks from the natural prompts. However, the adaptive attack lowers the perplexity of adversarial inputs well below natural prompts. Vicuna and Llama-2chat in Appendix H.2
Figure 3 : Figure 3: Attack perplexity under Llama-IT.We take natural prompts, prompts with adversarial suffixes, and prompts with adaptively trained adversarial suffixes and measure their log perplexity. The perplexity defense can perfectly distinguish the basic attacks from the natural prompts. However, the adaptive attack lowers the perplexity of adversarial inputs well below natural prompts. Vicuna and Llama-2chat in Appendix H.2CC-BY · From the PDF
Figure 4 : Figure4: Convergence over training(1) The left shows the convergence of the reward margin over 3 epochs of training. The models fine-tuned with their optimal learning rate and β factor are trained until saturation. Thus, models have been trained to be as robust as possible with DPO. (2) The right shows the defense success rate on natural prompts converging at 100% on models trained with only 10% of the dataset.
Figure 4 : Figure4: Convergence over training(1) The left shows the convergence of the reward margin over 3 epochs of training. The models fine-tuned with their optimal learning rate and β factor are trained until saturation. Thus, models have been trained to be as robust as possible with DPO. (2) The right shows the defense success rate on natural prompts converging at 100% on models trained with only 10% of the dataset.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 14

  1. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  2. P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318, 2024. · arXiv
  3. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. · arXiv
  4. S. Kotha, J. M. Springer, and A. Raghunathan. Understanding catastrophic forgetting in language models via implicit inference, 2023.
  5. B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. GeDi: Generative discriminator guided sequence generation. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4929-4952, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.424. URL https: //aclanthology.org/2021.findings-emnlp.424. · DOI · Paper page
  6. M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. · arXiv
  7. G. Nogara, F. Pierri, S. Cresci, L. Luceri, P. Törnberg, and S. Giordano. Toxic bias: Perspective api misreads german as more toxic. arXiv preprint arXiv:2312.12651, 2023. · arXiv
  8. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
  9. Y. M. Pa Pa, S. Tanizaki, T. Kou, M. van Eeten, K. Yoshioka, and T. Matsumoto. An attacker's dream? exploring the capabilities of chatgpt for developing malware. In Proceedings of the 16th Cyber Security Experimentation and Test Workshop, CSET '23, page 10-18, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400707889. doi: 10.1145/3607505.3607513. URL https://doi.org/10.1145/3607505.3607513. · DOI
  10. Y. Pan, L. Pan, W. Chen, P. Nakov, M.-Y. Kan, and W. Y. Wang. On the risk of misinformation pollution with large language models. arXiv preprint arXiv:2305.13661, 2023. · arXiv
  11. L. Rosenblatt, L. Piedras, and J. Wilkins. Critical perspectives: A benchmark revealing pitfalls in perspectiveapi. In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), pages 15-24, 2022.
  12. L. Sun, Y. Huang, H. Wang, S. Wu, Q. Zhang, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024. · arXiv
  13. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  14. K. Yang and D. Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.naacl-main.276. URL http://dx.doi.org/10. 18653/v1/2021.naacl-main.276. · DOI

Cited by

Sorted by citations

See all 20

Metadata

Authors
Kim, T., Kotha, S., & Raghunathan, A.
Affiliations
Carnegie Mellon University (US)
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.14725
arXiv
2403.14725
OpenAlex
W4393177814
Cited by
20
References
0
Citations by year
2026: 5 · 2025: 6 · 2024: 9
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
Keywords
jailbreaking, language model safety, safety enforcement, safety benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Testing the Limits of Jailbreaking Defenses with the Purple Problem. https://app.cochat.ai/papers/10.48550/arxiv.2403.14725
The paper
Kim, T., Kotha, S., & Raghunathan, A. (2024). Testing the Limits of Jailbreaking Defenses with the Purple Problem. arXiv. https://doi.org/10.48550/arxiv.2403.14725