SafeLawBench: Towards Safe Alignment of Large Language Models
Open accessAbstract
With the growing prevalence of large language models (LLMs), the safety of LLMs has raised significant concerns.However, there is still a lack of definitive standards for evaluating their safety due to the subjective nature of current safety benchmarks.To address this gap, we conducted the first exploration of LLMs' safety evaluation from a legal perspective by proposing the SafeLawBench benchmark.SafeLawBench categorizes safety risks into three levels based on legal standards, providing a systematic and comprehensive framework for evaluation.It comprises 24,860 multi-choice questions and 1,106 open-domain question-answering (QA) tasks.Our evaluation included 2 closed-source LLMs and 18 open-source LLMs using zero-shot and fewshot prompting, highlighting the safety features of each model.We also evaluated the LLMs' safety-related reasoning stability and refusal behavior.Additionally, we found that a majority voting mechanism can enhance model performance.Notably, even leading SOTA models like Claude-3.5-Sonnetand GPT-4o have not exceeded 80.5% accuracy in multi-choice tasks on SafeLawBench, while the average accuracy of 20 LLMs remains at 68.8%.We urge the community to prioritize research on the safety of LLMs.Our dataset and code are available.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SafeLawBench shows that even state-of-the-art models struggle to satisfy established legal principles.
“Legal alignment tests such as SafeLawBench (Cao et al., 2025) further show that even state-of-the-art models struggle to satisfy established legal principles”
From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Cao et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 41
- David Atkinson and Jacob Morrison. 2024. A legal risk taxonomy for generative artificial intelligence. arXiv preprint arXiv:2404.09479. · arXiv
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
- Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model eval- uation. In Proceedings of the Third Workshop on Nat- ural Language Generation, Evaluation, and Metrics (GEM), pages 339-352, Singapore. Association for Computational Linguistics.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anas- tasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chat- bot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st Inter- national Conference on Machine Learning, ICML'24. JMLR.org.
- Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen- hui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. · arXiv
- Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zam- brano, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Informa- tion Processing Systems, 36.
- Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948. · arXiv
- Sirui Han and Chao Xi. 2020. Financial regulation as interagency competition? the saga of venture capital rule-making in china. In Research Handbook on Asian Financial Law, pages 499-524.
- Mianqiu Huang, Xiaoran Liu, Shaojun Zhou, Mozhi Zhang, Chenkun Tan, Pengyu Wang, Qipeng Guo, Zhe Xu, Linyang Li, Zhikai Lei, et al. 2024. Longsafetybench: Long-context llms struggle with safety issues. arXiv preprint arXiv:2411.06899. · arXiv
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. · arXiv
- Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674. · arXiv
- Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Alex Qiu, Juntao Dai, and Yaodong Yang. 2024a. Aligner: Efficient alignment by learning to correct. Advances in Neural Information Processing Systems, 37:90853-90890.
- Jiaming Ji, Wenqi Chen, Kaile Wang, Donghai Hong, Sitong Fang, Boyuan Chen, Jiayi Zhou, Juntao Dai, Sirui Han, Yike Guo, et al. 2025a. Mitigating decep- tive alignment via self-monitoring. arXiv preprint arXiv:2505.18807. · arXiv
- Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Ji- ayi Zhou, Kaile Wang, et al. 2025b. Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models. arXiv preprint arXiv:2503.17682. · arXiv
- Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024b. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513. · arXiv
- Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023a. Beavertails: To- wards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023b. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852. · arXiv
- Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, et al. 2024c. Align any- thing: Training all-modality models to follow in- structions with language feedback. arXiv preprint arXiv:2412.15838. · arXiv
- Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, othersLucile Saulnier, Lélio Renard Lavaud, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825v1. · arXiv
- Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghal- lah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024b. Wildteaming at scale: From in-the- wild jailbreaks to (adversarially) safer language mod- els. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems.
- Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024a. Lexeval: A compre- hensive chinese legal benchmark for evaluating large language models. arXiv preprint arXiv:2409.20288. · arXiv
- Qinbin Li, Junyuan Hong, Chulin Xie, Jeffrey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, Bo Li, Bingsheng He, and Dawn Song. 2024c. Llm-pbe: Assessing data pri- vacy in large language models. Proc. VLDB Endow., 17(11):3201-3214.
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. · arXiv
- Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. 2024b. Are your llms capable of stable reasoning? arXiv preprint arXiv:2412.13147. Ministry of Justice. Ministry of justice of the people's republic of china. Accessed: 2025-01-18. · arXiv
- Junichiro Niimi. 2024. Dynamic sentiment analysis with local large language models using majority vot- ing: A study on factors affecting restaurant evalua- tion. arXiv preprint arXiv:2407.13069. · arXiv
- Aman Priyanshu, Yash Maurya, and Zuofei Hong. 2024. Ai governance and accountability: An analysis of an- thropic's claude. arXiv preprint arXiv:2407.01557. · arXiv
- Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety alignment should be made more than just a few tokens deep. In The Thir- teenth International Conference on Learning Repre- sentations. QwenTeam. 2024. Qwen2.5: A party of foundation models.
- Rodrigo Rodrigues do Carmo, Anísio Mendes Lacerda, and Daniel Hasan Dalip. 2017. A majority voting approach for sentiment analysis in short texts using topic models. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web, pages 449- 455.
- Ling Shi and Deyi Xiong. 2024. Criskeval: A chi- nese multi-level risk evaluation benchmark dataset for large language models. arXiv preprint arXiv:2406.04752. · arXiv
- Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024a. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. · arXiv
- Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024b. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. · arXiv
- Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387. · arXiv
- Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-not-answer: Evalu- ating safeguards in LLMs. In Findings of the Asso- ciation for Computational Linguistics: EACL 2024, pages 896-911, St. Julian's, Malta. Association for Computational Linguistics.
- Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. 2024. Sorry-bench: Systematically evaluating large lan- guage model safety refusal behaviors. arXiv preprint arXiv:2406.14598. · arXiv
- Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2024. On protecting the data privacy of large lan- guage models (llms): A survey. arXiv preprint arXiv:2403.05156. · arXiv
- Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, et al. 2025a. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arXiv:2502.06876. · arXiv
- Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. 2024. Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927. · arXiv
- Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2025. AIR-BENCH 2024: A safety benchmark based on regulation and policies specified risk categories. In The Thirteenth International Conference on Learning Representations.
- Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2). · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
Cited by
Sorted by citations
Survey on LLM Safety: Attacks, Defenses, Alignment, Metrics, and Guardrails
Jalan, P., Abishethvarman, V., Chandna, B., & Naseem, U. (2026) · Machine Learning · doi:10.1007/s10994-026-07060-8 · Cited by 11 · see citing works
Fang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
Assessing the impact of safety guardrails on large language models using irritability metrics
Teferra, B. G., Johny, N., Huang, S., Rueda, A., Kamaleddin, M. A., Dunlop, K., Zhang, Y., Jha, M., Sharma, D., & Bhat, V. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02333-3 · Cited by 5 · see citing works
Research on Aligning Large Models with Human Values Using Lora Fine-Tuning
Wang, Y., & Ma, N. (2025) · International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI) · doi:10.1109/cei66465.2025.11398511
AI as skills governance: reconfiguring authority and ethics in China’s TVET reform
Guoying, L. U., Han, S., & Guo, Y. (2026) · Asian Review of Political Economy · doi:10.1007/s44216-026-00082-y · Cited by 2 · see citing works
Self-citation
- Authors
- Cao, C., Zhu, H., Ji, J., Sun, Q., Zhu, Z., Yinyu, W., Dai, J., Yang, Y., Han, S., & Guo, Y.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Peking University (CN)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2025
- Published
- 2025
- Pages
- 14015-14048
- OpenAlex
- W4412888273
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 5 · 2025: 1
- FWCI
- 4.40
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hong Kong University of Science and Technology (R9911, T45-205/21-N)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language model safety, safety benchmark, open-domain question answering, refusal behavior, reasoning stability, majority voting, LLM safety evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
