CoChat
Sign Up
PreprintCited by 28Adversarial Robustness in Machine LearningTopic ModelingSecurity and Verification in Computing

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

Open access
Mazeika, Mantas, Phan, Long, Yin, Xuwang, Zou, Andy, Wang, Zifan, Mu, Norman, Sakhaee, Elham, Li, Nathaniel +4 more·2024·arXiv
DOI 10.48550/arxiv.2402.04249arXiv 2402.04249
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.04249

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The original HarmBench test set comprises 400 unique harmful queries.

    “The original HarmBench test set comprises 400 unique harmful queries.”

    From A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness (2026) · cites this paper as Mazeika et al., 2024

    Section 4.1 distinguishes the 400 textual behaviors in HarmBench as a whole from its test set, which contains 410 behaviors. Appendix B.2 confirms that the test set contains 410 behaviors, not 400. (LLM, conf=0.94)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Some judges were initially evaluated on 600 human validation samples and under light robustness conditions, such as HarmBench's fine-tuned LLaMA-2 13B model.

    “Some judges were initially evaluated on 600 human validation samples and under light robustness conditions, such as HarmBench's fine-tuned LLaMA-2 13B model (Mazeika et al., 2024)”

    From Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges (2025) · cites this paper as Mazeika et al., 2024

  • The JailbreakBench judge comparison dataset is commonly used and comprises 300 examples from AdvBench, HarmBench, and some original samples.

    “we run experiments on the commonly used judge comparison dataset from JailbreakBench (Chao et al., 2024), comprised of 300 examples from AdvBench (Zou et al., 2023), HarmBench (Mazeika et al., 2024) and some original samples.”

    From Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges (2025) · cites this paper as Mazeika et al., 2024

  • The authors of HarmBench explicitly test the classifier under some robustness conditions.

    “this is expected, as the authors of the paper explicitly test the classifier under some robustness conditions (Mazeika et al., 2024)”

    From Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges (2025) · cites this paper as Mazeika et al., 2024

  • Mazeika et al. (2024) presents a meta-evaluation study on the robustness of several judges to adversarial inputs.

    “Mazeika et al. (2024) presents a meta-evaluation study on the robustness of several judges to adversarial inputs.”

    From Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges (2025) · cites this paper as Mazeika et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 82

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Athalye, A., Carlini, N., and Wagner, D. Obfuscated gra- dients give a false sense of security: Circumventing de- fenses to adversarial examples. In International confer- ence on machine learning, pp. 274-283. PMLR, 2018.
  3. Bagdasaryan, E., Hsieh, T.-Y., Nassi, B., and Shmatikov, V. (ab)using images and sounds for indirect instruc- tion injection in multi-modal llms. arXiv preprint arXiv:2307.10490, 2023. · arXiv
  4. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. · arXiv
  5. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. · arXiv
  6. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. · arXiv
  7. Bailey, L., Ong, E., Russell, S., and Emmons, S. Image hi- jacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023. · arXiv
  8. Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evti- mov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724, 2023. · arXiv
  9. Brown, T. B., Mané, D., Roy, A., Abadi, M., and Gilmer, J. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017. · arXiv
  10. Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228, 2018. · arXiv
  11. Cao, B., Cao, Y., Lin, L., and Chen, J. Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348, 2023. · arXiv
  12. Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. Ieee, 2017.
  13. Carlini, N., Tramer, F., Dvijotham, K. D., Rice, L., Sun, M., and Kolter, J. Z. (certified!!) adversarial robustness for free! arXiv preprint arXiv:2206.10550, 2022. · arXiv
  14. Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Awadalla, A., Koh, P. W., Ippolito, D., Lee, K., Tramer, F., et al. Are aligned neural networks ad- versarially aligned? arXiv preprint arXiv:2306.15447, 2023. · arXiv
  15. Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., and Liang, P. S. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32, 2019.
  16. Casper, S., Lin, J., Kwon, J., Culp, G., and Hadfield-Menell, D. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442, 2023. · arXiv
  17. Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., and Mukhopadhyay, D. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology, 6(1):25-45, 2021.
  18. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries. arXiv preprint arXiv:2310.08419, 2023. · arXiv
  19. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  20. Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1310-1320. PMLR, 2019.
  21. Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
  22. Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020. · arXiv
  23. Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B. A., Fung, P., and Hoi, S. C. H. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023. · arXiv
  24. Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., and Liu, Y. Masterkey: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715, 2023. · arXiv
  25. Ebrahimi, J., Rao, A., Lowd, D., and Dou, D. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751, 2017. · arXiv
  26. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. · arXiv
  27. Ge, S., Zhou, C., Hou, R., Khabsa, M., Wang, Y.-C., Wang, Q., Han, J., and Mao, Y. Mart: Improving llm safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689, 2023. · arXiv
  28. Glukhov, D., Shumailov, I., Gal, Y., Papernot, N., and Papyan, V. Llm censorship: A machine learning chal- lenge or a computer security problem? arXiv preprint arXiv:2307.10719, 2023. · arXiv
  29. Gopal, A., Helm-Burger, N., Justen, L., Soice, E. H., Tzeng, T., Jeyapragasan, G., Grimm, S., Mueller, B., and Esvelt, K. M. Will releasing the weights of large language models grant widespread access to pandemic agents? arXiv preprint arXiv:2310.18233, 2023. · arXiv
  30. Goyal, S., Doddapaneni, S., Khapra, M. M., and Ravindran, B. A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys, 55(14s):1-39, 2023.
  31. Guo, C., Sablayrolles, A., Jégou, H., and Kiela, D. Gradient- based adversarial attacks against text transformers. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.- t. (eds.), Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pp. 5747-5757, Online and Punta Cana, Dominican Repub- lic, November 2021. Association for Computational Lin- guistics. doi: 10.18653/v1/2021.emnlp-main.464. · DOI
  32. Hazell, J. Large language models can be used to effec- tively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. · arXiv
  33. Hendrycks, D. and Mazeika, M. X-risk analysis for ai research. arXiv preprint arXiv:2206.05862, 2022. · arXiv
  34. Hendrycks, D., Mazeika, M., and Woodside, T. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001, 2023. · arXiv
  35. Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D. Catas- trophic jailbreak of open-source llms via exploiting gen- eration. arXiv preprint arXiv:2310.06987, 2023. · arXiv
  36. Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testug- gine, D., et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. · arXiv
  37. Iyyer, M., Wieting, J., Gimpel, K., and Zettlemoyer, L. Ad- versarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059, 2018. · arXiv
  38. Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-y., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for ad- versarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023. · arXiv
  39. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. · arXiv
  40. Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
  41. Jones, E., Dragan, A., Raghunathan, A., and Steinhardt, J. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381, 2023. · arXiv
  42. Kaufmann, M., Kang, D., Sun, Y., Basart, S., Yin, X., Mazeika, M., Arora, A., Dziedzic, A., Boenisch, F., Brown, T., et al. Testing robustness against unforeseen adversaries. arXiv preprint arXiv:1908.08016, 2019. · arXiv
  43. Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166, 2023. · arXiv
  44. Li, J., Ji, S., Du, T., Li, B., and Wang, T. Textbugger: Gen- erating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271, 2018. · arXiv
  45. Li, L., Ma, R., Guo, Q., Xue, X., and Qiu, X. Bert-attack: Adversarial attack against bert using bert. arXiv preprint arXiv:2004.09984, 2020. · arXiv
  46. Li, Y., Wei, F., Zhao, J., Zhang, C., and Zhang, H. Rain: Your language models can align themselves without fine- tuning. arXiv preprint arXiv:2309.07124, 2023. · arXiv
  47. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a. · arXiv
  48. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
  49. Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., and Gao, J. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020. · arXiv
  50. Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., and Liu, Y. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023c. · arXiv
  51. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
  52. Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T., Adler, S., Jiang, A., and Weng, L. A holistic approach to undesired content detection in the real world. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15009-15018, 2023.
  53. Mazeika, M., Zou, A., Mu, N., Phan, L., Wang, Z., Yu, C., Khoja, A., Jiang, F., O'Gara, A., Sakhaee, E., Xiang, Z., Rajabi, A., Hendrycks, D., Poovendran, R., Li, B., and Forsyth, D. Tdc 2023 (llm edition): The trojan detection challenge. In NeurIPS Competition Track, 2023.
  54. Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., Aggarwal, K., et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045, 2023. · arXiv
  55. Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909, 2020. · arXiv
  56. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
  57. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3419-3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. doi: 10.18653/v1/2022.emnlp-main. 225. · DOI · Paper page
  58. Qi, X., Huang, K., Panda, A., Wang, M., and Mittal, P. Visual adversarial examples jailbreak aligned large lan- guage models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, volume 1, 2023a.
  59. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023b. · arXiv
  60. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. · arXiv
  61. Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., and Cohen, J. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501, 2023. · arXiv
  62. Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
  63. Shah, R., Pour, S., Tagade, A., Casper, S., Rando, J., et al. Scalable and transferable black-box jailbreaks for lan- guage models via persona modulation. arXiv preprint arXiv:2311.03348, 2023. · arXiv
  64. Shayegani, E., Dong, Y., and Abu-Ghazaleh, N. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539, 2023. · arXiv
  65. Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. "do anything now": Characterizing and evaluating in-the- wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2023a. · arXiv
  66. Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.346. · DOI · Paper page
  67. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. · arXiv
  68. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
  69. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  70. Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. · arXiv
  71. Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. Adversarial glue: A multi- task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840, 2021. · arXiv
  72. Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235, 2023a. · arXiv
  73. Wang, Z., Pang, T., Du, C., Lin, M., Liu, W., and Yan, S. Better diffusion models further improve adversarial training. arXiv preprint arXiv:2302.04638, 2023b. · arXiv
  74. Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. · arXiv
  75. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. Taxonomy of risks posed by lan- guage models. In Proceedings of the 2022 ACM Confer- ence on Fairness, Accountability, and Transparency, pp. 214-229, 2022.
  76. Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. Hard prompts made easy: Gradient- based discrete optimization for prompt tuning and discov- ery. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  77. Xu, H., Ma, Y., Liu, H.-C., Deb, D., Liu, H., Tang, J.- L., and Jain, A. K. Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing, 17:151-178, 2020.
  78. Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. · arXiv
  79. Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., and Shi, W. How johnny can persuade llms to jailbreak them: Re- thinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024. · arXiv
  80. Zhang, W. E., Sheng, Q. Z., Alhazmi, A., and Li, C. Adver- sarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41, 2020.
  81. Zhou, A., Li, B., and Wang, H. Robust prompt optimiza- tion for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263, 2024. · arXiv
  82. Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., and Liu, J. Freelb: Enhanced adversarial training for natural lan- guage understanding. arXiv preprint arXiv:1909.11764, 2019. · arXiv

Cited by

Sorted by citations

Authors
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.04249
arXiv
2402.04249
OpenAlex
W4391632220
Cited by
28
References
0
Citations by year
2026: 3 · 2025: 19 · 2024: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
automated red teaming, HarmBench, large language models, evaluation framework, adversarial training, LLM robustness, malicious use of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review