CoChat
Sign Up
PreprintCited by 10Natural Language Processing TechniquesTopic ModelingReinforcement Learning in Robotics

RewardBench: Evaluating Reward Models for Language Modeling

Open access
Lambert, Nathan, Pyatkin, Valentina, Morrison, Jacob, Miranda, LJ, Lin, Bill Yuchen, Chandu, Khyathi, Dziri, Nouha, Kumar, Sachin +4 more·2024·arXiv
DOI 10.48550/arxiv.2403.13787arXiv 2403.13787
Open this paper in CoChat — free accountOpen PDF

Abstract

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.13787

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MAGICORE is complementary to and enhanced by progress in reward modeling, rather than constrained by it.

    “the community is actively advancing the performance of reward models evidenced by a benchmark for reward models (Lambert et al., 2024), MAGICORE is thus complementary to and enhanced by progress in reward modeling, rather than constrained by it.”

    From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Lambert et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, and Natasha Jaques. Moral foundations of large language models. arXiv preprint arXiv:2310.15337, 2023. · arXiv
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Techni- cal Report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  3. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. · arXiv
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. · arXiv
  5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. · arXiv
  6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073, 2022b. · arXiv
  7. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable LM 2 1.6B Technical Report. arXiv preprint arXiv:2402.17834, 2024. · arXiv
  8. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952. ISSN 00063444. URL http:// www.jstor.org/stable/2334029.
  9. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
  10. Joshua Clymer, Garrett Baker, Rohan Subramani, and Sam Wang. Generalization analogies (ge- nies): A testbed for generalizing ai oversight to hard-to-measure domains. arXiv preprint arXiv:2311.07723, 2023. · arXiv
  11. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting Language Models with High-quality Feedback. arXiv preprint arXiv:2310.01377, 2023. · arXiv
  12. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023. · arXiv
  13. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetun- ing of Quantized LLMs. arXiv preprint arXiv:2305.14314, 2023. · arXiv
  14. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward rAnked FineTuning for Generative Foundation Model Align- ment. arXiv preprint arXiv:2304.06767, 2023. · arXiv
  15. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
  16. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards mea- suring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388, 2023. · arXiv
  17. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V- usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008. PMLR, 17-23 Jul 2022.
  18. Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024a. · arXiv
  19. Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravin- skyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024b. · arXiv
  20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS, 2021.
  21. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a Changing Climate: Enhancing LM Adaptation with Tülu 2. arXiv preprint arXiv:2311.10702, 2023. · arXiv
  22. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. · arXiv
  23. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. ArXiv, abs/2310.00752, 2023b. URL https://api.semanticscholar.org/CorpusID:263334281. · arXiv
  24. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023c.
  25. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. arXiv preprint arXiv:2310.08491, 2023. · arXiv
  26. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024. · arXiv
  27. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv-2310, 2023.
  28. Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. · arXiv
  29. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023a. · arXiv
  30. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. arXiv preprint arXiv:2305.20050, 2023. · arXiv
  31. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023. · arXiv
  32. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. OctoPack: Instruc- tion Tuning Code Large Language Models. arXiv preprint arXiv:2308.07124, 2023. · arXiv
  33. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
  34. Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
  35. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. · arXiv
  36. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. · arXiv
  37. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022. URL https://arxiv.org/abs/2210. 01241. · arXiv
  38. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. arXiv preprint arXiv:2308.01263, 2023. · arXiv
  39. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  40. Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward (in-) consistency on rlhf. arXiv preprint arXiv:2309.16155, 2023. · arXiv
  41. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. · arXiv
  42. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008-3021.
  43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Founda- tion and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  44. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct Distillation of LM Alignment. arXiv preprint arXiv:2310.16944, 2023. · arXiv
  45. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv preprint arXiv:2308.13387, 2023. · arXiv
  46. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021. · arXiv
  47. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023. · arXiv
  48. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024b. · arXiv
  49. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating Large Language Models at Evaluating Instruction Following. arXiv preprint arXiv:2310.07641, 2023. · arXiv
  50. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685, 2023. · arXiv
  51. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023b. · arXiv
  52. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Table 11: REWARDBENCH results for the Chat Hard category. Icons refer to model types: Se- quence Classifier ( ), Direct Preference Optimization ( ), Custom Classifier ( ), Generative Model ( ), and a random model ( ). MTBench LLMBar LLMBar Adversarial Reward Model Avg. Hard Natural Neighbor GPTInst GPTOut Manual google/gemini-1.5-pro-0514 80.6 81.1 94.0 75.4 79.3 70.2 79.3 RLHFlow/ArmoRM-Llama3-8B-v0.1 76.8 86.5 93.0 67.9 77.2 66.0 69.6 openai/gpt-4-turbo-2024-04-09 75.4 86.5 97.0 53.0 80.4 74.5 76.1 openai/gpt-4-0125-preview 74.3 83.8 91.0 56.7 70.7 87.2 76.1 openai/gpt-4o-2024-05-13 70.4 78.4 91.0 50.7 71.7 74.5 69.6 Qwen/Qwen1.5-14B-Chat 70.2 67.6 71.0 83.6 62.0 46.8 71.7 Qwen/Qwen1.5-7B-Chat 69.1 64.9 65.0 81.3 59.8 53.2 80.4 upstage/SOLAR-10.7B-Instruct-v1.0 68.6 59.5 75.0 80.6 57.6 51.1 67.4 Qwen/Qwen1.5-72B-Chat 66.0 59.5 68.0 81.3 45.7 51.1 78.3 RLHFlow/pair-preference-model-LLaMA3-8B 65.8 75.7 89.0 53.0 62.0 68.1 50.0 openbmb/Eurus-RM-7b 65.6 78.4 93.0 53.0 55.4 63.8 54.3 sfairXC/FsfairX-LLaMA3-RM-v0.1 65.1 78.4 91.0 52.2 57.6 63.8 52.2 mistralai/Mixtral-8x7B-Instruct-v0.1 64.0 75.7 77.0 67.9 41.3 55.3 69.6 Qwen/Qwen1.5-MoE-A2.7B-Chat 63.2 54.1 59.0 72.4 53.3 57.4 78.3 Qwen/Qwen1.5-0.5B-Chat 62.9 45.9 58.0 75.4 65.2 48.9 60.9 HuggingFaceH4/zephyr-7b-beta 62.7 83.8 83.0 70.9 27.2 51.1 60.9 Qwen/Qwen1.5-4B-Chat 62.7 51.4 55.0 75.4 67.4 42.6 63.0 HuggingFaceH4/zephyr-7b-alpha 62.5 83.8 76.0 66.4 35.9 63.8 56.5 0-hero/Matter-0.1-7B-boost-DPO-preview 61.0 75.7 78.0 62.7 40.2 57.4 52.2 NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO 60.5 64.9 72.0 63.4 39.1 66.0 60.9 NousResearch/Nous-Hermes-2-Mistral-7B-DPO 60.5 75.7 80.0 55.2 45.7 55.3 56.5 allenai/tulu-2-dpo-70b 60.5 64.9 72.0 70.9 34.8 51.1 63.0 Anthropic/claude-3-opus-20240229 60.3 78.4 90.0 32.8 55.4 76.6 54.3 Qwen/Qwen1.5-1.8B-Chat 60.3 54.1 63.0 74.6 43.5 44.7 67.4 stabilityai/stablelm-zephyr-3b 60.1 86.5 74.0 81.3 18.5 36.2 54.3 meta-llama/Meta-Llama-3-70B-Instruct 58.9 81.1 83.0 32.5 57.6 71.3 55.4 stabilityai/stable-code-instruct-3b 58.6 51.4 53.0 79.9 38.0 48.9 65.2 allenai/tulu-2-dpo-13b 58.3 70.3 75.0 71.6 25.0 51.1 47.8 weqweasdas/RM-Mistral-7B 58.1 78.4 88.0 44.0 43.5 61.7 43.5 hendrydong/Mistral-RM-for-RAFT-GSHF-v0 57.9 81.1 91.0 46.3 40.2 59.6 34.8 0-hero/Matter-0.1-7B-DPO-preview 57.7 64.9 75.0 57.5 39.1 68.1 41.3 CohereForAI/c4ai-command-r-plus 57.6 74.3 84.0 26.9 63.0 70.2 52.2 Nexusflow/Starling-RM-34B 57.2 91.9 91.0 31.3 39.1 76.6 47.8 Anthropic/claude-3-sonnet-20240229 56.6 75.7 86.0 28.7 57.1 66.0 47.8 allenai/tulu-2-dpo-7b 56.1 67.6 70.0 70.9 25.0 40.4 52.2 openbmb/UltraRM-13b 55.5 75.7 82.0 42.5 43.5 51.1 47.8 HuggingFaceH4/starchat2-15b-v0.1 55.5 59.5 82.0 53.7 27.2 53.2 58.7 stabilityai/stablelm-2-12b-chat 55.5 64.9 70.0 73.1 18.5 44.7 50.0 jondurbin/bagel-dpo-34b-v0.5 55.0 48.6 69.0 73.9 25.0 34.0 56.5 PoLL/gpt-3.5-turbo-0125 claude-3-sonnet-20240229... 54.1 78.4 89.0 26.1 47.3 66.0 41.3 openbmb/Eurus-7b-kto 53.7 64.9 73.0 60.4 27.2 44.7 45.7 llm-blender/PairRM-hf 52.2 64.9 78.0 42.5 31.5 57.4 50.0 Anthropic/claude-3-haiku-20240307 52.0 67.6 77.0 33.6 46.7 61.7 39.1 allenai/OLMo-7B-Instruct 50.7 64.9 67.0 58.2 25.0 40.4 43.5 Ray2333/reward-model-Mistral-7B-instruct-Unified... 50.7 78.4 90.0 32.8 · arXiv

Cited by

Sorted by citations

Authors
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., & Hajishirzi, H.
Affiliations
University of Washington (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.13787
arXiv
2403.13787
OpenAlex
W4393063868
Cited by
5
References
0
Citations by year
2026: 1 · 2025: 3 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
reward models, benchmark dataset, reinforcement learning from human feedback, language model alignment, preference alignment, refusal behavior, instruction following, safety, direct preference optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review