CoChat
Sign Up
PreprintCited by 0Reinforcement Learning in RoboticsTopic ModelingDomain Adaptation and Few-Shot Learning

Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Open access
Liu, Michael Tian-Yue, Harrington, Anne, Darrell, Trevor, Bai, Yutong·2026·arXiv
DOI 10.48550/arxiv.2610.21179arXiv 2610.21179
Open this paper in CoChat — free accountOpen PDF

Abstract

Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper studies how the choice and combination of LLM judges affect Rubrics as Rewards (RaR) reinforcement learning, showing that judge identity alone substantially shifts trained policy quality. It proposes RaRE, a two-judge ensemble with soft-Likert confidence weighting and drop-on-disagreement masking, which outperforms every single-judge RaR variant on HealthBench and GPQA Diamond.

Key findings

  • Judge identity alone produces a substantial spread under a fixed rubric: 0.296–0.339 on English-only HealthBench (∼4.3 pp) across five judges, comparable to the gap between RL and the untrained base (0.257); GPQA Diamond single-judge spread is 0.345–0.406.
  • Naive averaging does not outperform the best single judge: Gemini+DeepSeek mean scores 0.307 on HealthBench, between Gemini (0.296) and DeepSeek (0.338); on GPQA Diamond, Gemini+DeepSeek mean is 0.363, close to constituents (0.360 Gemini, 0.345 DeepSeek).
  • RaRE’s best configuration outperforms every single-judge RaR variant: 0.364 vs. 0.339 best single judge on HealthBench, and 0.412 vs. 0.406 best single judge on GPQA Diamond. Per-theme analysis shows RaRE wins all seven HealthBench themes, and the dsmini ensemble wins 12 of 17 HealthBench categories outright.
  • Drop-on-disagreement is the most impactful mechanism: adding it to the naive Gemini+DeepSeek mean lifts HealthBench from 0.307 to 0.347 (+4.0 pp); removing it from the DeepSeek+4o-mini soft+drop configuration costs 3.4 pp (0.364 to 0.330). With drop fixed, switching from hard-Likert to soft-Likert lifts the DeepSeek+GPT-4o-mini pair from 0.295 to 0.364 (+6.9 pp).

Methods

  • The trainable policy is Qwen3-4B-Base, trained on RaR-Medicine and RaR-Science with GRPO using G=8 generations per prompt and per-group reward normalization; headline numbers use full-parameter fine-tuning. Five candidate single judges span proprietary, open-weight, and small-model regimes: gpt-4o-mini, DeepSeek-V3.1, Gemini-2.5-flash-lite, Qwen3-235B-Instruct, and gpt-oss-120b.
  • RaRE replaces the single training judge with a two-judge ensemble. Its soft-Likert reward reads top-5 logprobs at the rating-digit position to compute an expected rating and inverse-variance confidence per judge, then takes a confidence-weighted average. Its drop-on-disagreement mask excludes rollouts where the normalized rating spread exceeds τ by replacing their reward with the unmasked group mean, giving zero group-normalized advantage.
  • Evaluation uses the English-only subset of HealthBench for medicine with gpt-4o-mini as eval judge, and GPQA Diamond for science with disjoint 50/50 validation/test splits; GPQA results are means over 5 training seeds with 95% confidence intervals, and HealthBench uses bootstrapped 95% confidence intervals. Ablations use a 1,000-prompt HealthBench subset.

Limitations

  • The paper states that limited compute restricts experiments to a single base policy (Qwen3-4B-Base) and two domains (HealthBench and GPQA Diamond), so findings may not transfer to other policy families, model scales, or domains.
  • The paper states that HealthBench is evaluated with GPT-4o-mini, which is also used as a training judge, potentially introducing evaluation-judge bias; regrading with GLM-5.3-Flash keeps relative rankings similar and RaRE first. It also states that RaRE’s GPQA Diamond improvement over single-judge RaR is not statistically significant across 5 seeds.
  • The paper states that drop-on-disagreement discards dropped rollouts entirely rather than extracting partial signal, and that RaRE depends on inter-judge disagreement, providing little additional signal when judges largely agree. Evident from the text: the ensemble experiments use only two judges, and τ is tuned separately per domain (τ=0.30 for medicine, τ=0.40 for science).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2610.21179

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 8 cited sources checked · 0 supported · 1 flagged · accuracy not yet measured

  • Interpretationcitation

    Evaluation with GPT-4o-mini may introduce evaluation-judge bias since it is also used as a training judge.

    “We evaluate HealthBench with GPT-4o-mini, following the convention established by Arora et al. (2025) and Gunjal et al. (2025), which may introduce evaluation-judge bias since it is also used as a training judge.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Arora et al., 2025 source

      Section 8.1 (page unavailable) says Arora et al. adopted GPT-4.1—not GPT-4o-mini—as the default HealthBench grader. It notes that GPT-4.1 was used during prompt tuning, but does not say GPT-4o-mini was used as a training judge. The cited paper therefore does not establish the GPT-4o-mini grading convention attributed to it. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Gunjal et al., 2025 source
  • Premisecitation

    The broader LLM-as-a-verifier literature reports a similar finding in the evaluation setting, where explicit disagreement filtering matters more than the choice of any single judge.

    “The broader LLM-as-a-verifier literature (Lu et al., 2025;Singh et al., 2026;Kwok et al., 2026) reports a similar finding in the evaluation setting, where explicit disagreement filtering matters more than the choice of any single judge.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Lu et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Singh et al., 2026 source

      The Results section ('V1-Infer consistently outperforms pointwise self-verification'; page not provided) reports that pairwise comparisons outperform pointwise judgments across several models and benchmarks. This supports the general value of comparative verification, but the paper does not establish that explicit disagreement filtering matters more than the choice of any single judge. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Kwok et al., 2026 source

      Section 2 (page unavailable) reports that standard LM judges often tie when comparing complex solutions, and Section 4.2 (page unavailable) finds that repeated evaluations improve accuracy. These findings support the importance of handling uncertain comparisons, but the paper does not establish that explicit disagreement filtering matters more than the choice of any single judge. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    The HealthBench score from ensembling the naive mean of Gemini and DeepSeek's integer ratings (0.307) sits between the two constituent judges (0.296 Gemini, 0.338 DeepSeek).

    “The HealthBench score from ensembling the naive mean of Gemini and DeepSeek's integer ratings (0.307) sits between the two constituent judges (0.296 Gemini, 0.338 DeepSeek).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Adding drop-on-disagreement to the naive Gemini+DeepSeek mean ensemble lifts the score by +4.0 pp (0.307 to 0.347).

    “Adding it to the naive Gemini+DeepSeek mean ensemble lifts the score by +4.0 pp (0.307 to 0.347); removing it from the DeepSeek+4o-mini soft+drop configuration costs 3.4 pp (0.364 to 0.330).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.
  • Findingown data

    On GPQA Diamond, ensembling Gemini+DeepSeek naive mean (0.363) sits close to the two constituent judges (0.360 Gemini, 0.345 DeepSeek).

    “On GPQA Diamond, we see that ensembling Gemini+DeepSeek naive mean (0.363) also sits close to the two constituent judges (0.360 Gemini, 0.345 DeepSeek).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 42
  • Classified as probably not a substantial claim. 16
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Both top-logit and winner-takes-all are below the untrained base (0.266).

    “Both alternatives collapse (Table 5): top-logit scores 0.214 (-12 pp), winner-takes-all scores 0.198 (-14 pp), both below the untrained base, showing that confidence aggregation must account for the distribution of the judge's confidence.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Judge logprobs can be used to recover a judge's confidence in its rating.
    • Rests on an assumption: The top-5 logprobs at the rating-digit position capture the full distribution over ratings.

  • May combine two claims.

    Findingown data

    Composing soft-Likert with drop on the same pair (0.364) zeroes the high-disagreement rollouts and lets the soft-Likert reward on the rest become the strongest signal, exceeding the best single judge by +2.5 pp.

    “Composing soft-Likert with drop on the same pair (0.364) zeroes the high-disagreement rollouts and lets the soft-Likert reward on the rest become the strongest signal we have, exceeding the best single judge by +2.5 pp.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Judge logprobs can be used to recover a judge's confidence in its rating.
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.
    • Rests on an assumption: The top-5 logprobs at the rating-digit position capture the full distribution over ratings.

  • May combine two claims.

    Findingown data

    No single judge dominates; different judges have complementary strengths across HealthBench themes.

    “We break down performance of selected judges by HealthBench theme (Figure 4, showing each judge's delta in mean reward vs. the Qwen3-4B-Base baseline) and find that no single judge dominates; different judges have complementary strengths across themes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Naive averaging does not extract this gain: the mean of judges' ratings averages their individual scores and does not outperform the best single judge.

    “Naive averaging does not extract this gain: the mean of judges' ratings averages their individual scores and does not outperform the best single judge.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best RaRE configuration (soft + drop on DeepSeek+GPT-4o-mini) combines the strengths of its two constituent judges and achieves the best results in each theme across the board.

    “Furthermore, our best RaRE configuration (soft + drop on DeepSeek+GPT-4o-mini) combines the strengths of its two constituent judges and achieves the best results in each theme across the board.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    τ=0.15 is too aggressive (~14% of rollouts masked, gradient signal too sparse).

    “τ =0.15 is too aggressive (∼14% of rollouts masked, gradient signal too sparse), and τ =0.35 is too permissive (∼2% masked).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The LoRA dampening generalizes to RaRE: Gem+DS (mean+drop) scores 0.330 under full-FT vs. 0.295 under LoRA, a ~3 pp cost for substantially shorter outputs and lower training compute.

    “The LoRA dampening generalizes to RaRE: Gem+DS (mean+drop) scores 0.330 under full-FT vs. 0.295 under LoRA, a ∼3 pp cost for substantially shorter outputs and lower training compute (Appendix M).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    An explicit length penalty during training (4o-mini_lenpen) keeps responses short with no reward loss.

    “We explore two interventions that partially recover this: (i) an explicit length penalty during training (4o-mini_lenpen), which keeps responses short with no reward loss;”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The spread across judges on English-only HealthBench is 0.296-0.339 (~4.3 pp), comparable to the gap between RL and the untrained base (0.257).

    “The spread across judges (0.296-0.339 on English-only HealthBench, ∼4.3 pp) is comparable to the gap between RL and the untrained base (0.257).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Changing τ, removing the soft-Likert reward, or using alternative confidence weighting schemes all reduce the score substantially.

    “Here we show that each piece of this configuration is important: changing τ , removing the soft-Likert reward, or using alternative confidence weighting schemes all reduce the score substantially.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Adding drop-on-disagreement (mean + drop) to the Gemini+DeepSeek naive mean (0.347) lifts the score +0.8 pp above the best single judge by filtering the ~5% of rollouts where the two judges disagree most.

    “Adding drop-on-disagreement (mean + drop) to the Gemini+DeepSeek naive mean (0.347) lifts the score +0.8 pp above the best single judge by filtering the ∼5% of rollouts where the two judges disagree most.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.

  • May combine two claims.

    Findingown data

    τ=0.30 is best on the HealthBench 1K subset, with ~1.6 pp lift over no drop and scores dropping on either side.

    “We sweep τ on the Gem-ini+DeepSeek pair under the hard-Likert reward path and find τ =0.30 is best on the HealthBench 1K subset (Table 3), with ∼1.6 pp lift over no drop and scores dropping on either side.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With a stronger pair of judges, DeepSeek+4o-mini (0.400 4o-mini), the soft-Likert ensemble performs better (0.372), and adding drop-on-disagreement lifts score to 0.412, exceeding the best single judge.

    “However, with a stronger pair of judges, DeepSeek+4o-mini (0.400 4o-mini), the soft-Likert ensemble performs better (0.372), and adding drop-on-disagreement lifts score to 0.412, exceeding the best single judge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Judge logprobs can be used to recover a judge's confidence in its rating.
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.
    • Rests on an assumption: The top-5 logprobs at the rating-digit position capture the full distribution over ratings.

  • May combine two claims.

    Findingown data

    Single judges win on disjoint themes on HealthBench: DeepSeek wins communication, context_seeking, emergency_referrals, and global_health; GPT-4o-mini wins complex_responses, data_tasks, and hedging.

    “Figure 4 shows that single judges win on disjoint themes on HealthBench: DeepSeek wins communication, context_seeking, emergency_referrals, and global_health; GPT-4o-mini wins complex_responses, data_tasks, and hedging.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    τ=0.40 is the best disagreement threshold for the DeepSeek+4o-mini pair under the soft-Likert reward path on GPQA, filtering ~24% of rollouts.

    “We find that τ = 0.40 is the sweet spot, filtering ∼24% of rollouts.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.

  • May combine two claims.

    Findingown data

    The selected τ = 0.30 attains a score of 0.362 on the disjoint 50% test split, consistent with its validation score and with the headline medicine number.

    “The selected τ =0.30 attains a score of 0.362 on the disjoint 50% test split, consistent with its validation score and with the headline medicine number”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 50% validation split used for threshold selection is representative of the full HealthBench distribution.

  • May combine two claims.

    Findingown data

    In an ethics case study, GPT-4o-mini rated a response 4.99 despite a violation of an explicit rubric item, while DeepSeek rated it 1.96.

    “GPT-4o-mini rates it 4.99 despite the viola-tion, while DeepSeek rates it 1.96.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RaRE scores higher than GPT-4o under both evaluators at about one quarter of the API cost.

    “RaRE scores higher than GPT-4o under both evaluators at about one quarter of the API cost.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.

  • May combine two claims.

    Findingown data

    The gap between full-FT and LoRA is largest on the gemds drop030 ensemble and smaller for single-judge runs.

    “the gap is largest on the gemds drop030 ensemble and smaller for single-judge runs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The stricter τ = 0.25 is the worst setting, falling below even no filtering.

    “The stricter τ =0.25 is the worst setting, falling below even no filtering”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 50% validation split used for threshold selection is representative of the full HealthBench distribution.

  • May combine two claims.

    Findingown data

    In exploratory checks, English responses to Spanish prompts often received higher scores than Spanish responses, and translating rubrics into Spanish increased the average grade of Spanish responses.

    “In exploratory checks, English responses to Spanish prompts often received higher scores than Spanish responses, and translating rubrics into Spanish increased the average grade of Spanish responses.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Full-FT retro variants OOM'd before convergence at our hardware scale (80 GB H100s).

    “Full-FT retro variants OOM'd before convergence at our hardware scale (80 GB H100s).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The median per-batch reward latency is 1.01 s for a single GPT-4o-mini judge and 2.06 s for RaRE.

    “The median per-batch reward latency is 1.01 s for a single GPT-4o-mini judge and 2.06 s for RaRE, whose two judges are queried sequentially in our implementation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Single-judge RaR policies trained with GPT judges show the largest drops under GLM-5.3-Flash, while RaRE's score changes little.

    “Single-judge RaR policies trained with GPT judges show the largest drops under GLM-5.3-Flash,while RaRE's score changes little.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.

  • May combine two claims.

    Findingown data

    τ = 0.30 improves on no filtering by roughly 3.4 points, while both a stricter threshold (τ = 0.25) and a more permissive one (τ = 0.35) score lower.

    “τ =0.30 improves on no filtering by roughly 3.4 points, while both a stricter threshold (τ =0.25) and a more permissive one (τ =0.35) score lower.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 50% validation split used for threshold selection is representative of the full HealthBench distribution.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    RaRE's API cost splits into $0.99 for GPT-4o-mini and $3.47 for DeepSeek-V3.1.

    “RaRE's cost splits into $0.99 for GPT-4o-mini and $3.47 for DeepSeek-V3.1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Relative rankings of policies remain similar under GPT-4o-mini and GLM-5.3-Flash evaluators, and RaRE ranks first under both.

    “Relative rankings remain similar, and RaRE ranks first under both.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.

  • May combine two claims.

    Findingown data

    Regrading policy responses with GLM-5.3-Flash, a judge from a different model family, shows relative rankings remain similar, and RaRE ranks first under both judges.

    “Relative rankings remain similar, and RaRE ranks first under both judges.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Biology only has n=19 questions.

    “Biology only has n=19 questions”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A variant with GPT-4o-mini alone supplying the reward and two-judge disagreement filtering scores 34.2 on HealthBench, above single-judge GPT-4o-mini RaR (33.9) and below RaRE (36.4).

    “Table 8 shows that this variant scores 34.2 on HealthBench, above single-judge GPT-4o-mini RaR (33.9) and below RaRE (36.4).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.

  • May combine two claims.

    Findingown data

    Under the answer-order permutation protocol, RaRE's mean exceeds every single-judge RaR policy and is close to tuned Qwen3-4B, but the differences are not statistically significant.

    “Under this protocol, RaRE's mean exceeds every single-judge RaR policy and is close to tuned Qwen3-4B, but the differences are not statistically significant.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.
    • Rests on an assumption: The GPQA Diamond answer-order permutations adequately capture order sensitivity.

  • May combine two claims.

    Findingown data

    The soft-Likert reward provides additional gain when composed with the mask, but on its own does not outperform the best single judge on HealthBench.

    “The soft-Likert reward provides additional gain when composed with the mask, but on its own does not outperform the best single judge on HealthBench.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Judge logprobs can be used to recover a judge's confidence in its rating.
    • Rests on an assumption: The top-5 logprobs at the rating-digit position capture the full distribution over ratings.

  • May combine two claims.

    Findingown data

    It also wins 1 GPQA topic (biology) outright, with strong performances on the other 2 topics.

    “It also wins 1 GPQA topic (biology) outright, with strong performances on the other 2 topics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All configurations show a brief initial drop in length, followed by sustained expansion under single-judge RaR.

    “All configurations show a brief initial drop in length, followed by sustained expansion under single-judge RaR.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Across 25,600 training rollouts from four two-judge runs, 25.35% were masked, and GPT-4o-mini gave the higher rating in 99.61% of masked rollouts.

    “Across 25,600 training rollouts from four two-judge runs, 25.35% were masked, and GPT-4o-mini gave the higher rating in 99.61% of masked rollouts, suggesting that it applies rubric items more leniently than DeepSeek.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The best methods on RaR-Medicine and RaR-Science require 1 hour in total to train (8 H100-hours).

    “Our best methods on RaR-Medicine and RaR-Science require 1 hour in total to train (8 H100-hours).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    RaR-Science requires filtering a greater percentage of rollouts than medicine, suggesting that there is inherently much more judge disagreement on what a good science response should be.

    “Interestingly, RaR-Science requires filtering a greater percentage of rollouts than medicine, suggesting that there is inherently much more judge disagreement on what a good science response should be.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.

  • May combine two claims.

    Interpretationown data

    The judge side carries comparable signal and deserves comparable attention.

    “Our results suggest that the judge side carries comparable signal and deserves comparable attention.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Prior RaR work has focused almost exclusively on improving the rubric, treating the judge as fixed.

    “Prior RaR work has focused almost exclusively on improving the rubric, treating the judge as fixed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The drop-on-disagreement filter removes precisely those rollouts where judges disagree most, which could exclude cases where a single judge correctly flags a safety-relevant issue that others miss.

    “the drop-on-disagreement filter removes precisely those rollouts where judges disagree most, which could exclude cases where a single judge correctly flags a safety-relevant issue that others miss.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Rollouts where the two judges strongly disagree produce unreliable reward estimates.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    For GPQA Diamond, we train models on 5 seeds and report the mean score and 95% confidence intervals.

    “For GPQA Diamond, we train models on 5 seeds and report the mean score and 95% confidence intervals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Filtering with a second judge helps even when only one judge supplies the reward, and RaRE's soft-Likert ensemble reward adds further gains.

    “Filtering with a second judge therefore helps even when only one judge supplies the reward, and RaRE's soft-Likert ensemble reward adds further gains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The HealthBench English-only score is a valid measure of medical response quality.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The continuous reward signal allows finer-grained credit assignment.

    “since the continuous reward signal allows finer-grained credit assignment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    HealthBench provides bootstrapped 95% confidence intervals.

    “HealthBench provides bootstrapped 95% confidence intervals, stated in Appendix C.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We split GPQA Diamond into disjoint 50/50 validation/test splits for hyperparameter tuning, with results reported on the test split.

    “Furthermore, we split GPQA Diamond into disjoint 50/50 validation/test splits for purposes of hyperparameter tuning, with results reported on the test split.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Error bars in Figure 3 show the mean over five training seeds with 95% confidence intervals.

    “Error bars show the mean over five training seeds with 95% confidence intervals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The drop thresholds τ = 0.30 for Gem+DS and τ = 0.40 for DS+Mini were selected on a held-out validation set.

    “drop applies our drop-ondisagreement mask (τ = 0.30 for Gem+DS; τ = 0.40 for DS+Mini, selected on a held-out validation set).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Due to limited compute, experiments are restricted to a single base policy (Qwen3-4B-Base) evaluated on two domains: HealthBench (medicine) and GPQA Diamond (science), so findings may not transfer to other policy families, model scales, or other domains.

    “Due to limited compute, our experiments are restricted to a single base policy (Qwen3-4B-Base) evaluated on two domains: HealthBench (medicine) and GPQA Diamond (science), so our findings may not transfer to other policy families, model scales, or other domains.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    Comparisons for the Biology row are relatively unreliable.

    “making comparisons relatively unreliable.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Hisada et al. (2026) report that grading Japanese responses against English rubrics can penalize appropriate responses, and many rubric criteria require localization.

    “Hisada et al. (2026) report a related effect for Japanese: grading Japanese responses against English rubrics can penalize appropriate responses, and many rubric criteria require localization.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hisada et al. (2026)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

50 of 50

  • The paper relies on this without support.

    Judge logprobs can be used to recover a judge's confidence in its rating.

    “we read its top-5 logprobs to recover an expected rating E i and a per-judge confidence C i .”
    “we read the top-T logprobs at the rating-digit position to recover the full distribution over ratings.”

  • The paper relies on this without support.

    Rollouts where the two judges strongly disagree produce unreliable reward estimates.

    “Rollouts where the two judges strongly disagree produce unreliable reward estimates, where averaging does not reflect either judge's actual assessment.”

  • The paper relies on this without support.

    The top-5 logprobs at the rating-digit position capture the full distribution over ratings.

    “we read the top-T logprobs at the rating-digit position to recover the full distribution over ratings.”

  • The paper relies on this without support.

    The HealthBench English-only score is a valid measure of medical response quality.

    “Table 8 shows that this variant scores 34.2 on HealthBench, above single-judge GPT-4o-mini RaR (33.9) and below RaRE (36.4).”

  • The paper relies on this without support.

    The 50% validation split used for threshold selection is representative of the full HealthBench distribution.

    “Table 12 reports the disagreement-threshold sweep for the DeepSeek+4o-mini soft-Likert configuration on a 50% validation split of HealthBench.”

  • The paper relies on this without support.

    The GPQA Diamond answer-order permutations adequately capture order sensitivity.

    “Following Gunjal et al. (2025), we reevaluate each policy over ten trials, each using a different permutation of the four answer choices.”

  • The paper relies on this without support.

    The two-judge disagreement mask identifies rollouts where judges genuinely disagree about quality rather than noise.

    “Across 25,600 training rollouts from four two-judge runs, 25.35% were masked, and GPT-4o-mini gave the higher rating in 99.61% of masked rollouts, suggesting that it applies rubric items more leniently than DeepSeek.”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • LoRA and full-fine-tuning lengths disagree with Table 2

    Major

    The text gives mean lengths of 1,058 and 2,930 characters for mini-LoRA and full-FT mini, respectively. Table 2 gives 1,629 and 3,869 for those same configurations, while matching the stated rewards of 0.295 and 0.296.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Related

Authors
Liu, M. T.-Y., Harrington, A., Darrell, T., & Bai, Y.
Affiliations
University of California, Berkeley (US)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2610.21179
arXiv
2610.21179
OpenAlex
W7222545470
Semantic Scholar
8dee72f3905e3b391d822dd9a7dcd19cc9ed8978
Cited by
0
References
29
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Defense Science and Engineering Graduate
  • Multidisciplinary University Research Initiative
  • Office of Naval Research (N00014-21-1-2801)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
LLM-as-a-judge, reinforcement learning from human feedback, GPQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review