CoChat
Sign Up
Cited by 5Adversarial Robustness in Machine LearningSecurity and Verification in ComputingTopic Modeling

Sok: Evaluating Jailbreak Guardrails for Large Language Models

Open access
Wang, Xunguang, Ji, Zhenlan, Wang, Wenxuan, Li, Zongjie, Wu, Daoyuan, Wang, Shuai·2026·2026 IEEE Symposium on Security and Privacy (SP)
DOI 10.1109/sp63933.2026.00076
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have achieved remarkable progress, but their deployment has exposed critical vulnerabilities, particularly to jailbreak attacks that circumvent safety alignments. Guardrails--external defense mechanisms that monitor and control LLM interactions--have emerged as a promising solution. However, the current landscape of LLM guardrails is fragmented, lacking a unified taxonomy and comprehensive evaluation framework. In this Systematization of Knowledge (SoK) paper, we present the first holistic analysis of jailbreak guardrails for LLMs. We propose a novel, multi-dimensional taxonomy that categorizes guardrails along six key dimensions, and introduce a Security-Efficiency-Utility evaluation framework to assess their practical effectiveness. Through extensive analysis and experiments, we identify the strengths and limitations of existing guardrail approaches, provide insights into optimizing their defense mechanisms, and explore their universality across attack types. Our work offers a structured foundation for future research and development, aiming to guide the principled advancement and deployment of robust LLM guardrails. The code is available at https://github.com/xunguangwang/SoK4JailbreakGuardrails.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/sp63933.2026.00076

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Categories like 'Physical Harm' are easily intercepted, while subtle threats such as 'Sensitive Data Leakage' and 'Sophisticated Misinformation' often bypass the semantic filters.

    “Our granular analysis by risk category showed that whereas categories like”

    From Evaluation of NeMo Guardrails as a Firewall for User–LLM Interaction (2026) · cites this paper as [34]

    Section 4.1 (p. 5) says pre-processing guardrails may struggle with novel, implicit jailbreaks that conceal malicious content, supporting a general concern about evasion. It does not report that Physical Harm is easily intercepted, that Sensitive Data Leakage or Sophisticated Misinformation often bypass semantic filters, or that detection differs across those risk categories. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 91

  1. [1][1] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , 'Llama 2: Open foundation and fine-tuned chat models,' arXiv preprint arXiv:2307.09288 , 2023. - · arXiv
  2. [4][4] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. AlDahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , 'The llama 3 herd of models,' arXiv preprint arXiv:2407.21783 , 2024. - · arXiv
  3. [6][6] Qwen Team, 'Qwen2.5 technical report,' arXiv preprint arXiv:2412.15115 , 2024. - · arXiv
  4. [7][7] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, 'Universal and transferable adversarial attacks on aligned language models,' arXiv preprint arXiv:2307.15043 , 2023. - · arXiv
  5. [8][8] X. Liu, N. Xu, M. Chen, and C. Xiao, 'AutoDAN: Generating stealthy jailbreak prompts on aligned large language models,' in ICLR , 2024. -
  6. [9][9] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen, 'NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails,' in EMNLP: System Demonstrations , 2023. -
  7. [10][10] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine et al. , 'Llama Guard: LLM-based input-output safeguard for Human-AI conversations,' arXiv preprint arXiv:2312.06674 , 2023. - · arXiv
  8. [11][11] X. Wang, D. Wu, Z. Ji, Z. Li, P. Ma, S. Wang, Y. Li, Y. Liu, N. Liu, and J. Rahmel, 'SelfDefend: LLMs can defend themselves against jailbreaking in a practical manner,' in USENIX Security , 2025. -
  9. [12][12] S. Xhonneux, A. Sordoni, S. G¨ unnemann, G. Gidel, and L. Schwinn, 'Efficient adversarial training in LLMs with continuous attacks,' in NeurIPS , 2024. -
  10. [13][13] S. Zhang, Y. Zhai, K. Guo, H. Hu, S. Guo, Z. Fang, L. Zhao, C. Shen, C. Wang, and Q. Wang, 'JBShield: Defending large language models from jailbreak attacks through activated concept analysis and manipulation,' in USENIX Security , 2025. -
  11. [14][14] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu, 'Jailbreaking ChatGPT via prompt engineering: An empirical study,' arXiv preprint arXiv:2305.13860 , 2023. - · arXiv
  12. [15][15] A. Wei, N. Haghtalab, and J. Steinhardt, 'Jailbroken: How does LLM safety training fail?' in NeurIPS , 2023. -
  13. [16][16] Z. Wei, Y. Wang, and Y. Wang, 'Jailbreak and guard: Aligned language models with only few in-context demonstrations,' arXiv preprint arXiv:2310.06387 , 2023. - · arXiv
  14. [18][18] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, 'MASTERKEY: Automated jailbreaking of large language model chatbots,' in NDSS , 2024. -
  15. [19][19] C. Sitawarin, N. Mu, D. Wagner, and A. Araujo, 'PAL: Proxyguided black-box attack on large language models,' arXiv preprint arXiv:2402.09674 , 2024. - · arXiv
  16. [20][20] M. Andriushchenko, F. Croce, and N. Flammarion, 'Jailbreaking leading safety-aligned LLMs with simple adaptive attacks,' in ICLR , 2025. -
  17. [21][21] X. Jia, T. Pang, C. Du, Y. Huang, J. Gu, Y. Liu, X. Cao, and M. Lin, 'Improved techniques for optimization-based jailbreaking on large language models,' in ICLR , 2025. -
  18. [22][22] J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, S. Koyejo, H. Sleight, E. Jones, E. Perez, and M. Sharma, 'Best-of-N jailbreaking,' arXiv preprint arXiv:2412.03556 , 2024. - · arXiv
  19. [23][23] X. Chen, Y. Nie, W. Guo, and X. Zhang, 'When LLM meets DRL: Advancing jailbreaking efficiency via DRL-guided search,' in NeurIPS , 2024. -
  20. [24][24] E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, 'Red teaming language models with language models,' in EMNLP , 2022. -
  21. [25][25] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, 'Jailbreaking black box large language models in twenty queries,' arXiv preprint arXiv:2310.08419 , 2023. - · arXiv
  22. [26][26] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, 'Tree of attacks: Jailbreaking black-box LLMs automatically,' in NeurIPS , 2024. -
  23. [27][27] A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y. Tian, 'AdvPrompter: Fast adaptive adversarial prompting for LLMs,' in ICML , 2025. -
  24. [29][29] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, 'Tree of thoughts: Deliberate problem solving with large language models,' in NeurIPS , 2024. -
  25. [30][30] J. Yu, X. Lin, Z. Yu, and X. Xing, 'LLM-Fuzzer: Scaling assessment of large language model jailbreaks,' in USENIX Security , 2024. -
  26. [31][31] D. Handa, A. Chirmule, B. Gajera, and C. Baral, 'Jailbreaking proprietary large language models using word substitution cipher,' arXiv preprint arXiv:2402.10601 , 2024. - · arXiv
  27. [32][32] X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh, 'DrAttack: Prompt decomposition and reconstruction makes powerful LLMs jailbreakers,' in EMNLP Findings , 2024. -
  28. [33][33] Z. Chang, M. Li, Y. Liu, J. Wang, Q. Wang, and Y. Liu, 'Play guessing game with LLM: Indirect jailbreak attack with implicit clues,' in ACL , 2024. -
  29. [34][34] Y. Deng, W. Zhang, S. J. Pan, and L. Bing, 'Multilingual jailbreak challenges in large language models,' ICLR , 2024. -
  30. [35][35] Z.-X. Yong, C. Menghini, and S. H. Bach, 'Low-resource languages jailbreak GPT-4,' arXiv preprint arXiv:2310.02446 , 2023. - · arXiv
  31. [36][36] L. Shen, W. Tan, S. Chen, Y. Chen, J. Zhang, H. Xu, B. Zheng, P. Koehn, and D. Khashabi, 'The language barrier: Dissecting safety challenges of LLMs in multilingual contexts,' in ACL , 2024. -
  32. [37][37] J. Li, Y . Liu, C. Liu, L. Shi, X. Ren, Y . Zheng, Y . Liu, and Y . Xue, 'A cross-language investigation into jailbreak attacks in large language models,' arXiv preprint arXiv:2401.16765 , 2024. - · arXiv
  33. [38][38] Y. Yuan, W. Jiao, W. Wang, J. tse Huang, P. He, S. Shi, and Z. Tu, 'GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher,' in ICLR , 2024. -
  34. [39][39] J. Chu, Y. Liu, Z. Yang, X. Shen, M. Backes, and Y. Zhang, 'JailbreakRadar: Comprehensive assessment of jailbreak attacks against LLMs,' in ACL , 2025. -
  35. [40][40] Y. Meng, M. Xia, and D. Chen, 'SimPO: Simple preference optimization with a reference-free reward,' in NeurIPS , 2024. -
  36. [41][41] Z. Zhou, J. Xiang, H. Chen, Q. Liu, Z. Li, and S. Su, 'Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue,' arXiv preprint arXiv:2402.17262 , 2024. - · arXiv
  37. [42][42] X. Liu, L. Li, T. Xiang, F. Ye, L. Wei, W. Li, and N. Garcia, 'Imposter. AI: Adversarial attacks with hidden intentions towards aligned large language models,' arXiv preprint arXiv:2407.15399 , 2024. - · arXiv
  38. [43][43] N. Li, Z. Han, I. Steneker, W. Primack, R. Goodside, H. Zhang, Z. Wang, C. Menghini, and S. Yue, 'LLM defenses are not robust to multi-turn human jailbreaks yet,' arXiv preprint arXiv:2408.15221 , 2024. - · arXiv
  39. [44][44] X. Yang, X. Tang, S. Hu, and J. Han, 'Chain of attack: A semanticdriven contextual multi-turn attacker for LLM,' arXiv preprint arXiv:2405.05610 , 2024. - · arXiv
  40. [45][45] M. Russinovich, A. Salem, and R. Eldan, 'Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack,' in USENIX Security , 2025. -
  41. [46][46] Q. Ren, H. Li, D. Liu, Z. Xie, X. Lu, Y. Qiao, L. Sha, J. Yan, L. Ma, and J. Shao, 'Derail yourself: Multi-turn LLM jailbreak attack through self-discovered clues,' arXiv preprint arXiv:2410.10700 , 2024. - · arXiv
  42. [47][47] S. Rahman, L. Jiang, J. Shiffer, G. Liu, S. Issaka, M. R. Parvez, H. Palangi, K.-W. Chang, Y. Choi, and S. Gabriel, 'X-Teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents,' arXiv preprint arXiv:2504.13203 , 2025. - · arXiv
  43. [49][49] A. Zhou, B. Li, and H. Wang, 'Robust prompt optimization for defending language models against jailbreaking attacks,' in NeurIPS , 2024. -
  44. [50][50] Y. Mo, Y. Wang, Z. Wei, and Y. Wang, 'Fight back against jailbreaking via prompt adversarial tuning,' in NeurIPS , 2024. -
  45. [51][51] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, 'Training language models to follow instructions with human feedback,' in NeurIPS , 2022, pp. 27 730-27 744. -
  46. [52][52] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al. , 'Constitutional AI: Harmlessness from AI feedback,' arXiv preprint arXiv:2212.08073 , 2022. - · arXiv
  47. [53][53] W. Lu, Z. Zeng, J. Wang, Z. Lu, Z. Chen, H. Zhuang, and C. Chen, 'Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge,' arXiv preprint arXiv:2404.05880 , 2024. - · arXiv
  48. [54][54] W. Zhao, Z. Li, Y. Li, Y. Zhang, and J. Sun, 'Defending large language models against jailbreak attacks via layer-specific editing,' in EMNLP , 2024. -
  49. [55][55] Z. Zhang, J. Yang, P. Ke, F. Mi, H. Wang, and M. Huang, 'Defending large language models against jailbreaking attacks through goal prioritization,' in ACL , 2024. -
  50. [56][56] Y. Yuan, W. Jiao, W. Wang, J.-t. Huang, J. Xu, T. Liang, P. He, and Z. Tu, 'Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training,' in ACL , 2025. -
  51. [57][57] Y. Li, F. Wei, J. Zhao, C. Zhang, and H. Zhang, 'RAIN: Your language models can align themselves without finetuning,' in ICLR , 2024. -
  52. [58][58] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran, 'SafeDecoding: Defending against jailbreak attacks via safety-aware decoding,' in ACL , 2024. -
  53. [59][59] J. Ji, B. Chen, H. Lou, D. Hong, B. Zhang, X. Pan, T. Qiu, J. Dai, and Y. Yang, 'Aligner: Efficient alignment by learning to correct,' in NeurIPS , 2024. -
  54. [60][60] S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri, 'WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs,' in NeurIPS , 2024. -
  55. [61][61] A. Lees, V. Q. Tran, Y. Tay, J. Sorensen, J. Gupta, D. Metzler, and L. Vasserman, 'A new generation of perspective API: Efficient multilingual character-level transformers,' in KDD , 2022. -
  56. [62][62] T. Markov, C. Zhang, S. Agarwal, F. E. Nekoul, T. Lee, S. Adler, A. Jiang, and L. Weng, 'A holistic approach to undesired content detection in the real world,' in AAAI , 2023. -
  57. [63][63] M. Phute, A. Helbling, M. Hull, S. Peng, S. Szyller, C. Cornelius, and D. H. Chau, 'LLM Self Defense: By self examination, LLMs know they are being tricked,' arXiv preprint arXiv:2308.07308 , 2023. - · arXiv
  58. [64][64] G. Alon and M. Kamfonas, 'Detecting language model attacks with perplexity,' arXiv preprint arXiv:2308.14132 , 2023. - · arXiv
  59. [65][65] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, 'Baseline defenses for adversarial attacks against aligned language models,' arXiv preprint arXiv:2309.00614 , 2023. - · arXiv
  60. [67][67] A. Robey, E. Wong, H. Hassani, and G. J. Pappas, 'SmoothLLM: Defending large language models against jailbreaking attacks,' arXiv preprint arXiv:2310.03684 , 2023. - · arXiv
  61. [68][68] Y. Xie, M. Fang, R. Pi, and N. Gong, 'GradSafe: Detecting jailbreak prompts for LLMs via safety-critical gradient analysis,' in ACL , 2024. -
  62. [69][69] J. Ji, B. Hou, A. Robey, G. J. Pappas, H. Hassani, Y. Zhang, E. Wong, and S. Chang, 'Defending large language models against jailbreak attacks via semantic smoothing,' arXiv preprint arXiv:2402.16192 , 2024. - · arXiv
  63. [70][70] S. Goyal, M. Hira, S. Mishra, S. Goyal, A. Goel, N. Dadu, K. DB, S. Mehta, and N. Madaan, 'LLMGuard: Guarding against unsafe LLM behavior,' in AAAI , 2024. -
  64. [71][71] X. Hu, P.-Y. Chen, and T.-Y. Ho, 'Gradient Cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes,' in NeurIPS , 2024. -
  65. [72][72] Y. Zeng, Y. Wu, X. Zhang, H. Wang, and Q. Wu, 'AutoDefense: Multi-agent LLM defense against jailbreak attacks,' arXiv preprint arXiv:2403.04783 , 2024. - · arXiv
  66. [73][73] Z. Yuan, Z. Xiong, Y. Zeng, N. Yu, R. Jia, D. Song, and B. Li, 'RigorLLM: Resilient guardrails for large language models against undesired content,' in ICML , 2024. -
  67. [74][74] S. Ghosh, P. Varshney, E. Galinkin, and C. Parisien, 'Aegis: Online adaptive AI content safety moderation with ensemble of LLM experts,' arXiv preprint arXiv:2404.05993 , 2024. - · arXiv
  68. [77][77] A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks, 'Improving alignment and robustness with circuit breakers,' in NeurIPS , 2024. -
  69. [78][78] Z. Xiang, L. Zheng, Y. Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang et al. , 'GuardAgent: Safeguard LLM agents by a guard agent via knowledge-enabled reasoning,' in ICML , 2025. -
  70. [79][79] M. Kang and B. Li, ' R 2 -Guard: Robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning,' in ICLR , 2025. -
  71. [81][81] D. Schwartz, D. Bespalov, Z. Wang, N. Kulkarni, and Y. Qi, 'Graph of attacks with pruning: Optimizing stealthy jailbreak prompt generation for enhanced llm content moderation,' arXiv preprint arXiv:2501.18638 , 2025. - · arXiv
  72. [82][82] B. Manczak, E. Zemour, E. Lin, and V. Mugunthan, 'PrimeGuard: Safe and helpful LLMs through tuning-free routing,' arXiv preprint arXiv:2407.16318 , 2024. - · arXiv
  73. [83][83] W. Zeng, Y. Liu, R. Mullins, L. Peran, J. Fernandez, H. Harkous, K. Narasimhan, D. Proud, P. Kumar, B. Radharapu et al. , 'ShieldGemma: Generative AI content moderation based on Gemma,' arXiv preprint arXiv:2407.21772 , 2024. - · arXiv
  74. [84][84] J. Hu, Y. Dong, and X. Huang, 'Trust-oriented adaptive guardrails for large language models,' arXiv preprint arXiv:2408.08959 , 2024. - · arXiv
  75. [88][88] A. Peng, J. Michael, H. Sleight, E. Perez, and M. Sharma, 'Rapid response: Mitigating LLM jailbreaks with a few examples,' arXiv preprint arXiv:2411.07494 , 2024. - · arXiv
  76. [89][89] E. Galinkin and M. Sablotny, 'Improved large language model jailbreak detection via pretrained embeddings,' arXiv preprint arXiv:2412.01547 , 2024. - · arXiv
  77. [90][90] X. Hu, P.-Y. Chen, and T.-Y. Ho, 'Token Highlighter: Inspecting and mitigating jailbreak prompts for large language models,' in AAAI , 2025. -
  78. [91][91] S. Ghosh, P. Varshney, M. N. Sreedhar, A. Padmakumar, T. Rebedea, J. R. Varghese, and C. Parisien, 'Aegis2.0: A diverse ai safety dataset and risks taxonomy for alignment of LLM guardrails,' arXiv preprint arXiv:2501.09004 , 2025. - · arXiv
  79. [92][92] M. K. Rad, H. Nghiem, A. Luo, S. Wadhwa, M. Sorower, and S. Rawls, 'Refining input guardrails: Enhancing llm-as-a-judge efficiency through chain-of-thought fine-tuning and alignment,' arXiv preprint arXiv:2501.13080 , 2025. - · arXiv
  80. [93][93] Y. Liu, H. Gao, S. Zhai, J. Xia, T. Wu, Z. Xue, Y. Chen, K. Kawaguchi, J. Zhang, and B. Hooi, 'GuardReasoner: Towards reasoning-based LLM safeguards,' arXiv preprint arXiv:2501.18492 , 2025. - · arXiv
  81. [94][94] M. Sharma, M. Tong, J. Mu, J. Wei, J. Kruthoff, S. Goodfriend, E. Ong, A. Peng, R. Agarwal, C. Anil et al. , 'Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming,' arXiv preprint arXiv:2501.18837 , 2025. - · arXiv
  82. [95][95] S. Xiang, A. Zhang, Y. Cao, Y. Fan, and R. Chen, 'Beyond surfacelevel patterns: An essence-driven defense framework against jailbreak attacks in LLMs,' in ACL Findings , 2025. -
  83. [96][96] C. Yung, H. Huang, S. M. Erfani, and C. Leckie, 'Curvalid: Geometrically-guided adversarial prompt detection,' arXiv preprint arXiv:2503.03502 , 2025. - · arXiv
  84. [97][97] R. Pu, C. Li, R. Ha, L. Zhang, L. Qiu, and X. Zhang, 'MirrorShield: Towards universal defense against jailbreaks via entropy-guided mirror crafting,' arXiv preprint arXiv:2503.12931 , 2025. - · arXiv
  85. [98][98] X. Zhang, C. Zhang, T. Li, Y. Huang, X. Jia, M. Hu, J. Zhang, Y. Liu, S. Ma, and C. Shen, 'JailGuard: A universal detection framework for prompt-based attacks on LLM systems,' ACM Trans. Softw. Eng. Methodol. , 2025. -
  86. [99][99] B. Upadhayay, V. Behzadan et al. , 'X-Guard: Multilingual guard agent for content moderation,' arXiv preprint arXiv:2504.08848 , 2025. - · arXiv
  87. [100][100] J. Piet, X. Huang, D. Jacob, A. Chow, M. Alrashed, G. Zhao, Z. Hu, C. Sitawarin, B. Alomair, and D. Wagner, 'Jailbreaksovertime: Detecting jailbreak attacks under distribution shift,' arXiv preprint arXiv:2504.19440 , 2025. <!-- image --> Tested Guardrails Figure 6. The delay of guardrails on GPT-4-0125-Preview. AlpacaEval and OR-Bench are two normal prompt datasets and others are jailbreak attacks. - · arXiv
  88. [101][101] P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer et al. , 'JailbreakBench: An open robustness benchmark for jailbreaking large language models,' in NeurIPS , 2024. -
  89. [103][103] J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, 'OR-Bench: An over-refusal benchmark for large language models,' in ICML , 2025. -
  90. [104][104] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , 'GPT-4 technical report,' arXiv preprint arXiv:2303.08774 , 2023. - · arXiv
  91. [105][105] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, 'Fine-tuning aligned language models compromises safety, even when users do not intend to!' arXiv preprint arXiv:2310.03693 , 2023. - · arXiv

Cited by

Sorted by citations

Authors
Wang, X., Ji, Z., Wang, W., Li, Z., Wu, D., & Wang, S.
Affiliations
  • Hong Kong University of Science and Technology (HK)
  • Renmin University of China (CN)
  • Lingnan University (HK)
ORCID
Venue
2026 IEEE Symposium on Security and Privacy (SP)
Publisher
IEEE
Published
2026-05-18
Pages
39-58
ISSN
1081-6011 (ISSN-L)
DOI
10.1109/sp63933.2026.00076
OpenAlex
W4417353149
Cited by
5
References
0
Citations by year
2026: 5
FWCI
44.68
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
jailbreak attacks, LLM guardrails, large language models, safety alignment, defense mechanisms, evaluation framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review