BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
Open accessAbstract
In this paper, we introduce the BeaverTails dataset, aimed at fostering research on safety alignment in large language models (LLMs). This dataset uniquely separates annotations of helpfulness and harmlessness for question-answering pairs, thus offering distinct perspectives on these crucial attributes. In total, we have gathered safety meta-labels for 333,963 question-answer (QA) pairs and 361,903 pairs of expert comparison data for both the helpfulness and harmlessness metrics. We further showcase applications of BeaverTails in content moderation and reinforcement learning with human feedback (RLHF), emphasizing its potential for practical safety measures in LLMs. We believe this dataset provides vital resources for the community, contributing towards the safe development and deployment of LLMs. Our project page is available at the following URL: https://sites.google.com/view/pku-beavertails.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many real-world attacks exploit a separation between adversarial intent expressed in the prompt and actionable harm manifested only in the response.
“many real-world attacks exploit a separation between adversarial intent expressed in the prompt and actionable harm manifested only in the response.”
From Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats (2026) · cites this paper as [5]
Cites
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationMcIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
“This measure is independent of the harmlessness of the response, as it focuses solely on the quality, clarity, and relevance of the provided information” (Ji et al., 2024).”
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31 · see citing works
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Yu, E., Li, J., Liao, M., Wang, S., Gao, Z., Mi, F., & Hong, L. (2024) · arXiv · doi:10.48550/arxiv.2406.17626 · Cited by 7 · see citing works
“Red Teaming Language Models Ganguli et al. (2022) created a dataset called AnthropicRedTeam to identify, quantify, and mitigate potentially harmful outputs from LLMs. Ji et al. (2023) sampled from AnthropicRedTeam to build a dataset called BeaverTails for QA contexts.”
BackgroundMethodInfluentialR-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.10019 · Cited by 2 · see citing works
“Llama Guard presents superior performance in content moderation, with 91.50% of F1 in its internal test set, and approaches GPT-4 in public datasets such as Ope-nAI Mod (Markov et al., 2023) and BeaverTails-30k (Ji et al., 2024).”
BackgroundInfluentialSelf-citationVerifying Intent and Harm: A Unified Defense Against LLM-Generated Threats
Thota, P., Lei, Y., Thangaraj, S., Jonnalagadda, S. R., & Nilizadeh, S. (2026) · arXiv · doi:10.48550/arxiv.2606.26377
“LLMs have also been shown to generate harmful and toxic content across adversarial contexts spanning hate speech, harassment, dangerous instructions, and sexually explicit material [5], [6], [16].”
Influential
- Authors
- Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, C., Sun, R., Wang, Y., & Yang, Y.
- ORCID
- Yang, Yaodong
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 24678-24704
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2307.04657
- OpenAlex
- W7133189861
- Cited by
- 88
- References
- 1
- Citations by year
- 2026: 41 · 2025: 25 · 2024: 19 · 2023: 3
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- safety alignment, large language models, human preference data, preference-based alignment, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
