RewardBench 2: Advancing Reward Model Evaluation

“Experimental Setup We investigate how a reward model’s performance on our benchmark compares with its downstream performance when used in RLHF algorithms, particularly proximal policy optimization (PPO) [50] using the Open Instruct library.”

“Training for three epochs does not lead to strong benefits in any of the tested configurations (though it does occasionally slightly help, particularly at lower learning rates and in the Ties subset), even considering different training data and base models, so we drop training for three epochs…”

“Then, we aggregate the judgments for each set of 4 (or more, for ties) and score those ratings as though they were rewards—by giving the model a point for rating the correct response highest, and scoring two-way ties as partial credit of 0.5, three-way ties as 0.33, and four-way ties as 0.25…”

“With that data, a reward model r ∗ is trained to output a scalar value to predict the probability p ∗ of a prompt and completion falling in the chosen category, following a Bradley-Terry model of human preferences [17]: .”

“Using a pool of models strong at math, we sampled five completions per model at a temperature of 1.0 and used majority voting to select a gold answer, as is common practice [66, 67].”

“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”

“Recently, multiple works have studied how to use generative language models instead of classifiers [60, 61] or reward models that generate reasoning in addition to the standard classification probability [62, 63], particularly combined with scaling inference-time compute [64].”

“Rankings: In this setting, for a best-of-4 datapoint, we give the generative model a prompt and all four candidate completions and ask it to judge which is best.”

“For example, rejection sampling [16] uses pre-existing prompts to sample completions from a base model, which are then ranked by a reward model to create a high quality dataset for further training (used in [3, 4] and others).”

“Examples include aspect-conditioned models [57], high quality human datasets [41, 58], scaling [59], or debiasing data [55].”

“Then, we aggregate the judgments for each set of 4 (or more, for ties) and score those ratings as though they were rewards—by giving the model a point for rating the correct response highest, and scoring two-way ties as partial credit of 0.5, three-way ties as 0.33, and four-way ties as 0.25…”

“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”

“Using a pool of models strong at math, we sampled five completions per model at a temperature of 1.0 and used majority voting to select a gold answer, as is common practice [66, 67].”

“The scores on both benchmarks are less [34]) and reward model training (Skywork preference mix [38]).”

“Appendix B has further training details.”

“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”

“Using the Open-Instruct library [37] 5 , we trained a total of 120 reward models using the following approach (see Appendix H for hyperparameter tuning details): 1.”

“Recently, multiple works have studied how to use generative language models instead of classifiers [60, 61] or reward models that generate reasoning in addition to the standard classification probability [62, 63], particularly combined with scaling inference-time compute [64].”

“We find that our benchmark is strongly predictive of RM performance in best-of-N sampling, and we identify an important factor affecting a RM’s performance in RLHF: whether or not the policy model and RM come from the same model lineage.”

“Reinforcement learning methods like proximal policy optimization [50] and group relative policy optimization [52] train a policy by prompting it and using the reward model to score completions.”

“Examples include aspect-conditioned models [57], high quality human datasets [41, 58], scaling [59], or debiasing data [55].”

“(3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal [24, 25, 26, 27], process reward [28], or visual process reward [29, 30] models.”

“Safety is a nuanced task for LMs, so we draw on recent work on compliance over a variety of domains, CoCoNot [35], while taking steps to make the benchmark conservative in areas where user disagreements may exist on what a model should do.”

“For some domains, the base model overwhelmingly affects performance , and there is no clear trend for the data sources we explored.”

“RMs have been used extensively for RLHF training [5, 6], but also are used for online direct alignment algorithms [7], data filtering [8, 4], and inference-time scaling [9, 10].”

“For example, we see that the Skywork data is particularly helpful for focus and safety, while the Tulu data is better for factuality.”

“05 , following [51].”

“Recently, multiple works have studied how to use generative language models instead of classifiers [60, 61] or reward models that generate reasoning in addition to the standard classification probability [62, 63], particularly combined with scaling inference-time compute [64].”

“Existing Reward Models R EWARD B ENCH 2 is a challenging benchmark for top reward models, shown in Table 3 for the top twenty existing models where they are particularly challenged by the Instruction Following, Math, and Factuality subsets.”

“Hyperparameters : While common practice is to train reward models for only one epoch [1, 2, 3, 39, 40, 41], several recent works have found strong results with training for two or more [38, 41, 54, 55], so we experiment with training over 1, 2, and 3 epochs.”

“The CoCoNot taxonomy does not always encourage outright refusal, but rather, rubrics are nuanced to allow for partial refusals where appropriate.”

“RMs have been used extensively for RLHF training [5, 6], but also are used for online direct alignment algorithms [7], data filtering [8, 4], and inference-time scaling [9, 10].”

“(3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal [24, 25, 26, 27], process reward [28], or visual process reward [29, 30] models.”

“Examples include aspect-conditioned models [57], high quality human datasets [41, 58], scaling [59], or debiasing data [55].”

“…we aggregate the judgments for each set of 4 (or more, for ties) and score those ratings as though they were rewards—by giving the model a point for rating the correct response highest, and scoring two-way ties as partial credit of 0.5, three-way ties as 0.33, and four-way ties as 0.25 (random).”

“Additional evaluations included analysis of downstream scores when the RM is used within inference-time methods such as best-of-N (BoN) sampling [13] and also training with RLHF [14].”

“Best-of-N (BoN) sampling is often included as a baseline relative to RLHF methods [5, 53], but it is better seen as an inference-time scaling method where the weights of the generating model are not changed.”

“(2) Specific new attributes to test include multilinguality [20], agentic systems (e.g., web agents [22] or retrieval augmented generation [23]), typos [19], and others [18].”

“Hyperparameters : While common practice is to train reward models for only one epoch [1, 2, 3, 39, 40, 41], several recent works have found strong results with training for two or more [38, 41, 54, 55], so we experiment with training over 1, 2, and 3 epochs.”

“Additionally, while benchmarks whose chosen-rejected splits are determined by human or LM pairwise preferences have some benefits, there is subjectivity in the preferences they prescribe as optimization targets [31, 32].”

“Additionally, while benchmarks whose chosen-rejected splits are determined by human or LM pairwise preferences have some benefits, there is subjectivity in the preferences they prescribe as optimization targets [31, 32].”

“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”

“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”

“(2) Specific new attributes to test include multilinguality [20], agentic systems (e.g., web agents [22] or retrieval augmented generation [23]), typos [19], and others [18].”

“Recent work has explored if accuracy-based RM benchmarks are correlated with downstream performance at all [21, 42]; [42] finds that in addition to overall RM accuracy, the variance in scores that a RM assigns to a policy model’s outputs to a given prompt also affects an RM’s performance in RLHF…”

“We additionally see that larger reward models perform better on the benchmark; this is to be expected, as their base models are stronger.”

“RMs have been used extensively for RLHF training [5, 6], but also are used for online direct alignment algorithms [7], data filtering [8, 4], and inference-time scaling [9, 10].”

“Hyperparameters : While common practice is to train reward models for only one epoch [1, 2, 3, 39, 40, 41], several recent works have found strong results with training for two or more [38, 41, 54, 55], so we experiment with training over 1, 2, and 3 epochs.”

“For example, we see that the Skywork data is particularly helpful for focus and safety, while the Tulu data is better for factuality.”

“(3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal [24, 25, 26, 27], process reward [28], or visual process reward [29, 30] models.”

“Combining both data sources improves average performance , outperforming training on either dataset alone across all base models.”

“We follow LLMBar [36] (and the Chat-Hard subdomain of the first R EWARD B ENCH ) and rewrite human prompts using a language model to introduce slight differences, which then induce objectively incorrect, off-topic, and/or generally unresponsive “rejected” completions that are misaligned in some way…”

“We append a constraint taken from the taxonomy of a new instruction-following benchmark, IFEval-OOD [34], to each prompt, manually ensuring relevance (more details in Appendix E.2) We use verifier functions to evaluate adherence to the constraint, and constructed each data instance by combining 1…”

“RMs have been used extensively for RLHF training [5, 6], but also are used for online direct alignment algorithms [7], data filtering [8, 4], and inference-time scaling [9, 10].”

“Additional evaluations included analysis of downstream scores when the RM is used within inference-time methods such as best-of-N (BoN) sampling [13] and also training with RLHF [14].”

“This involves four stages: prompt sourcing, where most of our prompts are unreleased human-written queries from WildChat [15]; prompt quality and domain annotation using classifiers; completion generation, where we aim for diversity while ensuring we construct both “right” and “wrong” completions;…”

“RMs have been used extensively for RLHF training [5, 6], but also are used for online direct alignment algorithms [7], data filtering [8, 4], and inference-time scaling [9, 10].”

“Hyperparameters : While common practice is to train reward models for only one epoch [1, 2, 3, 39, 40, 41], several recent works have found strong results with training for two or more [38, 41, 54, 55], so we experiment with training over 1, 2, and 3 epochs.”

“(3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal [24, 25, 26, 27], process reward [28], or visual process reward [29, 30] models.”

“The Math, Safety, and Focus domains are new datasets inspired by improving upon the Math, Safety, and Chat-Hard domains of the original R EWARD B ENCH [11], respectively, whereas Factuality, Precise IF, and Ties are designed to test additional capabilities of RMs not captured in R EWARD B ENCH .”

“We find that our benchmark is strongly predictive of RM performance in best-of-N sampling, and we identify an important factor affecting a RM’s performance in RLHF: whether or not the policy model and RM come from the same model lineage.”

“Recent work has explored if accuracy-based RM benchmarks are correlated with downstream performance at all [21, 42]; [42] finds that in addition to overall RM accuracy, the variance in scores that a RM assigns to a policy model’s outputs to a given prompt also affects an RM’s performance in RLHF…”

“Recently, multiple works have studied how to use generative language models instead of classifiers [60, 61] or reward models that generate reasoning in addition to the standard classification probability [62, 63], particularly combined with scaling inference-time compute [64].”

“(3) Benchmarks testing different modalities or structures of reward modeling include those for multimodal [24, 25, 26, 27], process reward [28], or visual process reward [29, 30] models.”

“IfEval and PopQA are relatively less correlated with R EWARD B ENCH 2, but this mirrors their lower correlation with other downstream tasks, as shown in Figure 7.”

“Publicly reported results indicate this system attains high accuracy on RewardBench 2 (Malik et al., 2025) via an ensemble-then-arbiter design, consistent with our use as an auditable reward signal within ATLAS.”

“Existing reward models trained on general human preferences fail to evaluate essential domain-specific dimensions, such as pragmatic correctness, vocabulary diversity, and task-oriented scaffolding logic [24, 25].”