The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness
Open accessAbstract
As Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications, their safety concerns become critical areas of NLP research.This has resulted in the development of various LLM defense strategies.Unfortunately, despite the shared goal of improving the safety of LLMs, the evaluation suites across various research works are disjoint and lack diverse inputs to ensure accurate and precise evaluation estimates.Furthermore, the important factor of 'over-defensiveness' on the safe inputs has largely remained overlooked.Addressing these limitations, this paper presents a systematic evaluation, comparison, and analysis of various LLM defense strategies over both 'safety' and 'over-defensiveness'.To this end, we compile a large and diverse collection of safe and unsafe prompts, design precise evaluation methodology, and study the efficacy of various LLM defense strategies on multiple state-of-the-art LLMs.Our work reveals a number of crucial findings that we believe will pave the way and also facilitate further research in the critical area of improving the safety of LLMs.WARNING: This paper may contain toxic and offensive model responses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ICD can reduce the ASR of Llama-2 against transferable GCG from 21% to 0%.
“ICD can reduce the ASR of Llama-2 [41] against transferable GCG from 21% to 0%”
From Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2026) · cites this paper as [41]
Section 4.1 reports a 21% baseline rate of unsafe responses for LLaMA on the paper’s unsafe prompts. Section 3.2 describes in-context exemplars, and the results table in Appendix C reports a lower rate with exemplars. The paper does not report an ICD result against transferable GCG or a reduction from 21% to 0%; its baseline is not identified as GCG attack success rate. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 18
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. · arXiv
- Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. Safety-tuned llamas: Lessons from improving the safety of large lan- guage models that follow instructions. arXiv preprint arXiv:2309.07875. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023. Defending against alignment-breaking at- tacks via robustly aligned llm. arXiv preprint arXiv:2309.14348. · arXiv
- Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srini- vasan, Tianyi Zhou, Heng Huang, et al. 2023. Al- pagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701. · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
- Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308. · arXiv
- Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for ad- versarial attacks against aligned language models. arXiv preprint arXiv:2309.00614. · arXiv
- Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Ag- garwal, Hamid Palangi, Guoqing Zheng, Corby Ros- set, Hamed Khanpour, and Ahmed Awadallah. 2023. Orca 2: Teaching small language models how to rea- son. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774. · arXiv
- Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. · arXiv
- Abhinav Rao, Sachin Vashistha, Atharva Naik, So- mak Aditya, and Monojit Choudhury. 2023. Trick- ing llms into disobedience: Understanding, ana- lyzing, and preventing jailbreaks. arXiv preprint arXiv:2305.14965. · arXiv
- Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral. 2023. Accelerating llm inference by enabling intermediate layer decoding. arXiv preprint arXiv:2310.18581. · arXiv
- Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387. · arXiv
- Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large lan- guage models to follow complex instructions. arXiv preprint arXiv:2304.12244. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. · arXiv
- Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrik- son. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. · arXiv
Cited by
Sorted by citations
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 209 · see citing works
Hate Speech Detection Using Large Language Models: A Comprehensive Review
Albladi, A., Islam, M., Das, A., Bigonah, M., Zhang, Z., Jamshidi, F., Rahgouy, M., Raychawdhary, N., Marghitu, D., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3532397 · Cited by 72 · see citing works
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51 · see citing works
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 70 · see citing works
SoK: Prompt Hacking of Large Language Models
Rababah, B., Wu, S. T., Kwiatkowski, M. R., Leung, C. K.-S., & Akçora, C. G. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825103 · Cited by 20 · see citing works
Jain, B., Pawar, P., Gada, D., Patwa, T., Kanani, P., Patil, D., & Kurup, L. (2025) · International Research Journal of Multidisciplinary Technovation · doi:10.54392/irjmt2523 · Cited by 10 · see citing works
Survey on LLM Safety: Attacks, Defenses, Alignment, Metrics, and Guardrails
Jalan, P., Abishethvarman, V., Chandna, B., & Naseem, U. (2026) · Machine Learning · doi:10.1007/s10994-026-07060-8 · Cited by 11 · see citing works
Wang, H., Qi, Z., Wang, N., Yu, Z., Li, Z., & Lv, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227721 · Cited by 2 · see citing works
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 3 · see citing works
Attacks and Defenses on Aligned Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_13
PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization
Liu, A., Tang, L., Pan, T., Yin, Y., Wang, B., & Yang, A. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210045
Adversarial Attacks on Large Language Models: The Prompt Injection Approach
Siri, H., Vijayshree, V. M., & Shetty, J. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-4452-3_17
FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-Shots Learning
Ren, Y., Chen, J., & Zhang, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_47
- Authors
- Varshney, N., Dolin, P., Seth, A., & Baral, C.
- Affiliations
- Arizona State University (US)
- ORCID
- Baral, Chitta
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 13111-13128
- OpenAlex
- W4402670087
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 2
- FWCI
- 3.75
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language model safety, prompt evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
