SafetyBench: Evaluating the Safety of Large Language Models
Open accessAbstract
With the rapid development of Large Language Models (LLMs), increasing attention has been paid to their safety concerns. Consequently, evaluating the safety of LLMs has become an essential task for facilitating the broad applications of LLMs. Nevertheless, the absence of comprehensive safety evaluation benchmarks poses a significant impediment to effectively assess and enhance the safety of LLMs. In this work, we present SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data, facilitating the evaluation in both languages. Our extensive tests over 25 popular Chinese and English LLMs in both zero-shot and few-shot settings reveal a substantial performance advantage for GPT-4 over its counterparts, and there is still significant room for improving the safety of current LLMs. We also demonstrate that the measured safety understanding abilities in SafetyBench are correlated with safety generation abilities. Data and evaluation guidelines are available at \url{https://github.com/thu-coai/SafetyBench}{https://github.com/thu-coai/SafetyBench}. Submission entrance and leaderboard are available at \url{https://llmbench.ai/safety}{https://llmbench.ai/safety}.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SafetyBench is a benchmark specifically designed to test security performance of a range of popular Chinese and English LLMs.
“Zhang et al. [254] have introduced SafetyBench, a benchmark specifically designed to test the security performance of a range of popular Chinese and English LLMs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [254]
Results of SafetyBench evaluation reveal substantial security flaws in current LLMs.
“The results of this evaluation reveal substantial security flaws in current LLMs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [254]
Larger and newer models still showed stigma and failed to respond appropriately.
“We found larger and newer models (with, in theory, better safety filtering and tuning [114, 157]) still showed stigma (Fig. 1 and 6) and failed to respond appropriately (Fig. 4).”
From Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. (2025) · cites this paper as [157]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · see citing works
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · see citing works
Self-citationAGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 1× · see citing works
“Next, we will introduce the category breakdown and the data collection process in detail.”
MethodToxicity in ChatGPT: Analyzing Persona-assigned Language Models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2304.05335 · Cited by 178 · see citing works
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works
Self-citationShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety DetectorsPreprint
Zhang, Z., Lu, Y.-D., Ma, J.-Y., Zhang, D., Li, R., Ke, P., Sun, H., Sha, L., Sui, Z.-F., Wang, H.-N., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2402.16444 · Cited by 73 · see citing works
Self-citationLongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. (2023) · arXiv · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 17
- Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941-1955, Online. Association for Computational Linguistics. · DOI · Paper page
- Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. 2022. COLD: A benchmark for Chinese offensive language detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 11580-11599, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
- Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
- Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335.
- Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situ- ated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics. · DOI · Paper page
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning AI with shared human values. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021b. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. CoRR, abs/2312.06674. · DOI · Paper page
- Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, and William Yang Wang. 2022. SafeText: A benchmark for exploring physical safety in language models. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 2407-2421, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. · DOI
- Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. SCRUPLES: A corpus of community ethical judg- ments on 32, 000 real-life anecdotes. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2021, Virtual Event, Febru- ary 2-9, 2021, pages 13470-13479. AAAI Press. · DOI · Paper page
- Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086-2105, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
- Ethan Perez, Saffron Huang, H. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red team- ing language models with language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3419-3448. Association for Computa- tional Linguistics. · DOI · Paper page
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
- Zhexin Zhang, Jiaxin Wen, and Minlie Huang. 2023. ETHICIST: targeted training data extraction through loss smoothed soft prompting and calibrated confi- dence estimation. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 12674-12687. Asso- ciation for Computational Linguistics. · DOI
- Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. 2022. Towards identifying social bias in di- alog systems: Framework, dataset, and benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3576-3591, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics. · DOI
- Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 3755-3773, Dublin, Ireland. Association for Computational Linguistics. · DOI
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“[240] have introduced SafetyBench, a benchmark specifically designed to test the security performance of a range of popular Chinese and English LLMs.”
BackgroundInfluentialMoore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204 · see citing works
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.10019 · Cited by 2 · see citing works
“(Richards, 2023), Voyager (Wang et al., 2023a) and OpenHands (Wang et al., 2024), can autonomously complete user-specified tasks with LLMs as controllers (Zhou et al., 2023b; Lin et al., 2023a).”
Self-citationEvaluation of NeMo Guardrails as a Firewall for User–LLM Interaction
Azambuja, A. J., Guilherme, M., Castro, J. V. F. D., Lima, J. P. D. O., Oliveira, L. B., & Soares, A. D. S. (2026) · Future Internet · doi:10.3390/fi18050252 · Cited by 2 · see citing works
Kong, S., Yang, X., Wei, Y., Wang, Z., Tang, H., Qin, J., Lan, S., Wang, Y., Bai, J., Chen, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01252 · Cited by 3 · see citing works
- Authors
- Zhang, Z., Lei, L., Wu, L., Sun, R., Huang, Y., Long, C., Liu, X., Lei, X., Tang, J., & Huang, M.
- Affiliations
- Tsinghua University (CN)
- Northwest Minzu University (CN)
- Peking University (CN)
- China Mobile Research Institute (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2309.07045
- OpenAlex
- W4386755570
- Cited by
- 19
- References
- 7
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 8 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Fund for Distinguished Young Scholars
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, LLM safety, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
