CoChat
Sign Up
PreprintCited by 3Ethics and Social Impacts of AIAdversarial Robustness in Machine LearningSafety Systems Engineering in Autonomy

AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies

Open access
Zeng, Yi, Yang, Yu, Zhou, Andy, Tan, Jeffrey Ziwei, Tu, Yuheng, Mai, Yifan, Klyman, Kevin, Pan, Minzhou +4 more·2024·arXiv
DOI 10.48550/arxiv.2407.17436arXiv 2407.17436
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models (FMs) provide societal benefits but also amplify risks. Governments, companies, and researchers have proposed regulatory frameworks, acceptable use policies, and safety benchmarks in response. However, existing public benchmarks often define safety categories based on previous literature, intuitions, or common sense, leading to disjointed sets of categories for risks specified in recent regulations and policies, which makes it challenging to evaluate and compare FMs across these benchmarks. To bridge this gap, we introduce AIR-Bench 2024, the first AI safety benchmark aligned with emerging government regulations and company policies, following the regulation-based safety categories grounded in our AI risks study, AIR 2024. AIR 2024 decomposes 8 government regulations and 16 company policies into a four-tiered safety taxonomy with 314 granular risk categories in the lowest tier. AIR-Bench 2024 contains 5,694 diverse prompts spanning these categories, with manual curation and human auditing to ensure quality. We evaluate leading language models on AIR-Bench 2024, uncovering insights into their alignment with specified safety concerns. By bridging the gap between public benchmarks and practical AI risks, AIR-Bench 2024 provides a foundation for assessing model safety across jurisdictions, fostering the development of safer and more responsible AI systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.17436

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Safety policies governing LLM-based agents are often complex and encoded in lengthy regulation documents or corporate policy handbooks, making it difficult to systematically extract, verify, and enforce rules across different platforms.

    “(2) Safety policies governing these agents are often complex and encoded in lengthy regulation documents (e.g. EU AI Act (Act, 2024)) or corporate policy handbooks (GitLab, 2025), making it difficult to systematically extract, verify, and enforce rules across different platforms (Zeng et al., 2024).”

    From ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning (2025) · cites this paper as Zeng et al., 2024

    Section 2.1 (p. 2) says the taxonomy was built by manually extracting risk categories from government regulations and corporate policies, and the Introduction (p. 3) identifies gaps when existing benchmarks are mapped to those categories. This supports the need to organize and assess policy-derived risks, but the paper does not establish that lengthy, complex documents make it difficult to extract, verify, and enforce rules across agent platforms. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing guardrails for LLMs are not applicable to LLM-based agents due to the complex and dynamic nature of agents.

    “More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents.”

    From ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning (2025) · cites this paper as Zeng et al., 2024

    The refusal study (section 4.3, p. 8) identifies gaps between current safety guardrails and AI regulatory requirements. I searched for agent-specific guardrails and the claimed complexity or dynamism of agents, but found no passage saying that LLM guardrails are inapplicable to agents. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • GuardAgent focuses solely on textual space and relies on the model's internal knowledge rather than explicitly enforcing compliance with external safety policies and regulations, limiting its effectiveness in real-world applications.

    “While GuardAgent (Xiang et al., 2024) preliminarily explores the challenge of guardrailing LLM agents with another LLM agent, it focus solely on textual space and still relies on the model's internal knowledge rather than explicitly enforcing compliance with external safety policies and regulations (Zeng et al., 2024), limiting its effectiveness in real-world applications.”

    From ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning (2025) · cites this paper as Zeng et al., 2024

    The discussion of existing benchmarks (p. 3) identifies gaps in their coverage of regulatory risks, and the EU AI Act case study (p. 8) identifies gaps between models’ refusal behavior and regulatory requirements. Neither passage addresses GuardAgent. Searches for GuardAgent, textual-only operation, and reliance on internal knowledge found no passage supporting the specific claim about it. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. [3]Markus Anderljung, Joslyn Barnhart, Jade Leung, Anton Korinek, Cullen O'Keefe, Jess Whit- tlestone, Shahar Avin, Miles Brundage, Justin Bullock, Duncan Cass-Beggs, et al. Frontier ai regulation: Managing emerging risks to public safety. arXiv preprint arXiv:2307.03718, 2023. · arXiv
  3. [8]Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing ai risks in an era of rapid progress. arXiv preprint arXiv:2310.17688, 2023. · arXiv
  4. [10]Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318, 2024. · arXiv
  5. [22]Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak chal- lenges in large language models. arXiv preprint arXiv:2310.06474, 2023. · arXiv
  6. [26]Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. https://docs.cohere.com/docs/usage-guidelines https://docs.cohere.com/docs/usage-guidelines https://www.chinalawtranslate.com/en/algorithms/ https://www.chinalawtranslate.com/en/algorithms/ https://www.chinalawtranslate.com/en/deep-synthesis/ https://www.chinalawtranslate.com/en/deep-synthesis/ https://www.chinalawtranslate.com/en/generative-ai-interim/ https://www.chinalawtranslate.com/en/generative-ai-interim/ https://www.tc260.org.cn/upload/2024-03-01/1709282398070082466.pdf https://www.tc260.org.cn/upload/2024-03-01/1709282398070082466.pdf https://github.com/DeepSeek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL https://github.com/DeepSeek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL https://chat.deepseek.com/downloads/DeepSeek%20User%20Agreement.html https://chat.deepseek.com/downloads/DeepSeek%20User%20Agreement.html https://platform.DeepSeek.com/downloads/DeepSeek%20Open%20Platform%20Terms%20of%20Service.html https://platform.DeepSeek.com/downloads/DeepSeek%20Open%20Platform%20Terms%20of%20Service.html https://platform.DeepSeek.com/downloads/DeepSeek%20Open%20Platform%20Terms%20of%20Service.html https://artificialintelligenceact.eu/ https://artificialintelligenceact.eu/ https://data.europa.eu/eli/reg/2016/679/oj https://data.europa.eu/eli/reg/2016/679/oj · arXiv
  7. [27]Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. · arXiv
  8. [28]Gemini Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
  9. [35]Saghar Hosseini, Hamid Palangi, and Ahmed Hassan Awadallah. An empirical study of metrics to measure representational harms in pre-trained language models. arXiv preprint arXiv:2301.09211, 2023. · arXiv
  10. [37]Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044, 2024. · arXiv
  11. [38]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. · arXiv
  12. [39]Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. arXiv preprint arXiv:2310.17389, 2023. · arXiv
  13. [40]Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. · arXiv
  14. [44]Ministry of Science and Technology of Cina. Scientific and technological ethics review regula- tion (trial). www.gov.cn/zhengce/zhengceku/202310/content_6908045.htm, 2023.
  15. [53]Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024.
  16. [54]Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023. · arXiv
  17. [62]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
  18. [63]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  19. [65]Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698, 2023. · arXiv
  20. [66]Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023. · arXiv
  21. [67]Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, et al. Sociotech- nical safety evaluation of generative ai systems. arXiv preprint arXiv:2310.11986, 2023. https://openai.com/research/gpt-4v-system-card https://openai.com/research/gpt-4v-system-card https://web.archive.org/web/20240109122522/https:/openai.com/policies/usage-policies https://web.archive.org/web/20240109122522/https:/openai.com/policies/usage-policies https://cdn.openai.com/spec/model-spec-2024-05-08.html https://cdn.openai.com/spec/model-spec-2024-05-08.html https://openai.com/policies/usage-policies https://openai.com/policies/usage-policies https://stability.ai/use-policy · arXiv
  22. [68]Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214–229, 2022.
  23. [69]Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598, 2024. · arXiv
  24. [70]Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705, 2023. · arXiv
  25. [71]Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023. · arXiv
  26. [72]Yi Zeng, Kevin Klyman, Andy Zhou, Yu Yang, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. Ai risk categorization decoded (air 2024): From government regulations to corporate policies. arXiv preprint arXiv:2406.17864, 2024. · arXiv
  27. [73]Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024. · arXiv
  28. [74]Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. · arXiv

Cited by

Sorted by citations

Authors
Zeng, Y., Yang, Y., Zhou, A., Tan, J. Z., Tu, Y., Mai, Y., Klyman, K., Pan, M., Jia, R., Song, D., Liang, P., & Li, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.17436
arXiv
2407.17436
OpenAlex
W6947993656
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
foundation models, AI regulation, AI risk taxonomy, acceptable use policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review