CoChat
Sign Up
PreprintCited by 2Natural Language Processing TechniquesTopic ModelingReliability and Agreement in Measurement

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

Open access
Norman, Justin D., Rivera, Michael U., Hughes, D. Alex·2026·arXiv
DOI 10.48550/arxiv.2606.19544arXiv 2606.19544
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper reports the largest systematic evaluation of LLM-as-a-Judge to date, covering 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench under agreement, consistency, and bias-audit protocols (118 runs, ~541,000 judgments). It finds that exact-match agreement systematically overstates chance-corrected discriminative ability, judge rankings are benchmark-dependent, and high test-retest reliability can coexist with severe position bias.

Key findings

  • Kappa deflation is universal: exact match overstates Cohen’s κ by 33.8–41.3 percentage points on MT-Bench across all 21 judges (cohort mean 38.6 pp); even the best chance-corrected judge, Gemini 3.1 Pro, has EM = 0.849 vs. κ = 0.511 (33.8 pp gap). Mean deflation is 23.7 pp on JudgeBench and 10.4 pp on RewardBench.
  • Judge rankings are non-transferable: rankings shift by up to 14 positions across benchmarks (abstract/conclusion; Table 7 reports a maximum shift of 15 for Llama 3.3 70B). JudgeBench spreads the same 21 judges over a 60.4 pp κ range vs. MT-Bench’s 13.5 pp, a 4.5× wider spread.
  • The consistency-bias paradox is instantiated by two judges with test-retest > 0.95 and position bias > 0.10: Qwen 3 8B (test-retest 0.992, position bias 0.192) and Gemini 2.5 Flash (0.988, 0.125); Qwen 3 8B also has JudgeBench κ = 0.289.
  • Verbosity bias is small across the cohort: all 21 judges register < 0.011 on MT-Bench under a single pairwise rubric, with the largest value GPT-4o-mini at 0.010 and 17 of 21 below 0.005.

Methods

  • Evaluated 21 general-purpose LLM judges from nine providers, grouped into three capability tiers and spanning 8B to over 100B parameters, on MT-Bench (2,391 pairwise comparisons), JudgeBench (350 items), and RewardBench (2,981 chosen-versus-rejected pairs).
  • Used three protocols: agreement (single judgment vs. human label; Cohen’s κ, Krippendorff’s α, tie-excluded exact match), consistency (N = 3–5 independent runs per item, AB+BA orderings, response caching disabled; test-retest, self-consistency, position flip rate), and bias audit (AB+BA plus response-length analysis; position bias, verbosity bias). All evaluations used temperature 0; total 118 runs and ~541,000 judgments.
  • Formulated seven hypotheses before frontier-model evaluation and analyzed kappa deflation Δκ = EM − κ, cross-benchmark rank shifts, and provider/tier patterns.

Limitations

  • Stated: benchmark coverage is limited to three English-language, text-only benchmarks; multilingual and multimodal judging are not characterized. All runs occurred in a five-week window in March–April 2026, so results are a snapshot and hosted-model endpoints may drift. Findings use a single pairwise rubric and fixed operationalization, so the small verbosity-bias result should not be generalized. Calibration metrics (ECE, Brier score) were deferred because most providers do not expose token-level logprobs. Costs are list prices only. Reasoning traces were suppressed for thinking models, so results do not characterize thinking-on configurations.
  • Stated: the seven hypotheses were not externally pre-registered; one was refuted (H5) and two were partially confirmed (H2, H4).
  • Evident from the text (appendix): agreement values are population statistics over full benchmarks, bias-audit values are point estimates over paired AB+BA evaluations, and consistency metrics are point estimates over N = 3–5 replicates, reported without confidence intervals. RewardBench required per-item position randomization with fixed seed 42; under the standard fixed ordering, κ degenerates to 0.000 for every judge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.19544

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 14 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    Minimax M2.7 rises 11 positions from MT-Bench to JudgeBench (MT#16 → JB#5).

    “Minimax M2.7 rises 11 (MT#16 → JB#5)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).
  • Findingown data

    Within-family heterogeneity in position bias is large: Gemini 2.5 Pro (0.002) and Gemini 2.5 Flash (0.125) differ by a factor of 70.

    “Within-family heterogeneity is large: Gemini 2.5 Pro (0.002) and Gemini 2.5 Flash (0.125) differ by a factor of 70.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Three judges show ≥2.4× degradation in position flip rate from MT-Bench to JudgeBench.

    “Three judges show ≥ 2.4× degradation (highlighted orange).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).
  • Findingown data

    High test-retest reliability (≥0.95) coexists with severe position bias (>0.10) for Qwen 3 8B and Gemini 2.5 Flash.

    “high test-retest reliability (≥ 0.95) coexists with severe position bias (> 0.10) for Qwen 3 8B and Gemini 2.5 Flash”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).
  • Findingown data

    Only Claude Opus 4.6 and Gemini 3.1 Pro hold a stable top-3 position across all three benchmarks.

    “Only Claude Opus 4.6 and Gemini 3.1 Pro hold a stable top-3 position across all three benchmarks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).
  • May combine two claims. 40
  • The quote only partly states this claim. 4
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Premisecitation

    Judge validation in practice relies heavily on exact match, and established judge benchmarks privilege raw agreement as the headline validation metric.

    “Despite these challenges, judge validation continues to rely heavily on exact match.Established judge benchmarks-MT-Bench (Zheng et al., 2023), RewardBench (Lambert et al., 2025), JudgeBench (Tan et al., 2025)-and recent successors (Malik et al., 2026;Whitehouse et al., 2025;Jiang et al., 2025) privilege raw agreement as the headline validation metric.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Zheng et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lambert et al., 2025 source

      Section 4.2, “REWARDBENCH Scoring” (page unavailable), says RewardBench scores models by whether they rank the chosen response above the rejected one, then aggregates accuracy into a final score. That supports the claim that RewardBench makes agreement with preference labels its headline metric. The paper does not establish that judge validation generally relies heavily on exact match, or that the other named benchmarks use the same metric. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Tan et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Exact match does not correct for agreement expected by chance and systematically overstates discriminative ability.

    “This family of metrics does not correct for chance (Cohen, 1960;Hayes and Krippendorff, 2007) and overstates discriminative ability (Guerdan et al., 2025;Han et al., 2025;Collot et al., 2025).”
    • Also stated in: § 2.1 LLM Judge Metrics and Definitions · plain
      “Despite being widely used, exact match does not correct for agreement expected by chance, and as a result exact match is sensitive to the label distribution of the underlying benchmark.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Cohen, 1960 source
    • cites Hayes and Krippendorff, 2007 source
    • cites Guerdan et al., 2025 source

      Section 4.2 (page unavailable) reports that forced-choice agreement metrics can mis-rank judge systems and lead to suboptimal selections when humans and judges resolve rating indeterminacy differently. The appendix discussion of Figure 34 says categorical metrics perform similarly to other metrics for fully specified tasks or symmetric selection effects. This supports a conditional limitation of exact-match-style metrics, but not the unqualified claim that exact match systematically overstates discriminative ability. The chance-correction point is attributed to the other citations, not Guerdan et al. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Every judge in the study exhibits substantial kappa deflation on MT-Bench: exact match overstates chance-corrected agreement by between 33.8 and 41.3 percentage points across the 21 models, with a cohort mean of 38.6 pp.

    “Every judge in our study exhibits substantial kappa deflation on MT-Bench: exact match overstates chance-corrected agreement by between 33.8 and 41.3 percentage points across the 21 models, with a cohort mean of 38.6 pp.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Judge rankings shift across benchmarks, with models shifting by as many as 14 positions; more than half (11 of 21) shift, and ten of twenty-one exhibit a maximum pairwise rank shift of at least four positions.

    “judge rankings shift by up to 14 positions across benchmarks”
    • Also stated in: § 5.2 Limits of Single-Benchmark Validation · plain
      “Only two judges (Claude Opus 4.6 and Gemini 3.1 Pro) hold a top-three position across all three benchmarks; ten of twenty-one exhibit a maximum pairwise rank shift of at least four positions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Within OpenAI models, generational progression is visible on JudgeBench: κ rises from 0.309 for GPT-4o to 0.487 for GPT-4.1, and to 0.606 for GPT-5.4; on MT-Bench the corresponding values are 0.451, 0.451, and 0.457, making the progression nearly invisible.

    “Within-OpenAI developed models, generational progression is visible: κ rises from 0.309 for GPT-4o to 0.487 for GPT-4.1, and rises to 0.606 for GPT-5.4.MT-Bench's compressed scale makes the same generational progression nearly invisible: the corresponding MT-Bench values are 0.451, 0.451, and 0.457.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    H4: MT-Bench ceiling effect: full spread 13.5 pp; top-10 spread 6.5 pp. Partially confirmed (top-10 close to threshold).

    “H4 MT-Bench ceiling effect. | κ spread across all | Full spread 13.5 pp; top-10 spread | Partially con- | §4.4”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingreasoning

    The magnitude of kappa deflation varies systematically with the benchmark's label distribution: MT-Bench mean 38.6 pp, JudgeBench mean 23.7 pp (range 8.1-38.5), RewardBench mean 10.4 pp (range 5.9-21.3).

    “The magnitude of deflation varies systematically with the benchmark's label distribution.MT-Bench, with a balanced A/B/Tie distribution, produces the largest mean deflation (38.6 pp); JudgeBench's pairwise correctness-labeled items produce a mean of 23.7 pp (range 8.1 to 38.5); RewardBench's binary chosen-versus-rejected pairs produce 10.4 pp (range 5.9 to 21.3).”
    • Also stated in: § 5.1 Why Exact Match Misleads · boosted
      “The cohort-mean ∆ κ contracts from 38.6 pp on MT-Bench to 23.7 pp on JudgeBench and 10.2 pp on RewardBench, tracking the shift from balanced ternary to imbalanced binary labels exactly as Cohen's correction predicts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The three Anthropic judges average κ = 0.770 on JudgeBench with an average position bias of 0.020, the strongest joint performance on hard items and the lowest cohort-level bias of any provider.

    “The three Anthropic judges average κ = 0.770 on JudgeBench (Opus 4.6 0.875, Sonnet 4.6 0.782, Haiku 4.5 0.653) with an average position bias of 0.020, the strongest joint performance on hard items and the lowest cohort-level bias of any provider.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Two judges, Qwen 3 8B (test-retest 0.992, position bias 0.192) and Gemini 2.5 Flash (test-retest 0.988, position bias 0.125), exhibit high test-retest reliability (>0.95) together with severe position bias (>0.10), instantiating the consistency-bias paradox.

    “high test-retest reliability (> 0.95) coexists with severe position bias (> 0.10) in two production-deployed judges (instantiating a consistency-bias paradox)”
    • Also stated in: § 4.6 Consistency Falls on Hard Benchmarks · plain
      “Two judges, Qwen 3 8B (test-retest 0.992, position bias 0.192) and Gemini 2.5 Flash (test-retest 0.988, position bias 0.125) generate such paradoxical results.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Of the seven a priori predictions, three are fully confirmed (H1, H3, H7), one is confirmed for the explicitly-stated subset (H6), two are partially confirmed (H2, H4), and one is refuted (H5).

    “Of the seven a priori predictions, three are fully confirmed (H1, H3, H7), one is confirmed for the explicitly-stated subset (H6), two are partially confirmed (H2, H4), and one is refuted (H5).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    H6: Position flip rate degrades from MT-Bench to JudgeBench: 7 of 16 consistency-cohort judges show ≥1.5×. Confirmed for 7 of 16.

    “H6 Position flip rate de- | ≥ 1.5× flip-rate in- | 7 of 16 consistency-cohort judges | Confirmed for | §4.6”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    H7: Consistency-bias paradox is instantiated: 2 judges (Qwen 3 8B: test-retest 0.992, position bias 0.192; Gemini 2.5 Flash: 0.988, 0.125). Confirmed.

    “H7 Consistency-bias paradox is | ≥ 1 judge with test- | 2 judges (Qwen 3 8B: 0.992, 0.192; | Confirmed | §4.7”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    All ten frontier-generation (Tier 3) judges exhibit Δκ ≥ 30 pp on MT-Bench, with observed range [33.8, 41.2] pp; H1 confirmed.

    “The deflation is universal across capability tiers: all ten frontier-generation (Tier 3) judges exhibit ∆ κ ≥ 30 pp, supporting H1.”
    • Also stated in: § G Use of AI Assistance · plain
      “H1 Kappa deflation persists | ∆κ ≥ 30 pp for all 10 | Range [33.8, 41.2] pp; all 10 fron- | Confirmed | §4.1”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    Under per-item position-randomized evaluation, all 20 evaluated judges on RewardBench produce κ values in the range [0.616, 0.898], with the top five being Gemini 3.1 Pro (0.898), GPT-oss 120B (0.880), Claude Opus 4.6 (0.879), GPT-5.4 (0.879), and Claude Haiku 4.5 (0.873).

    “Under per-item position-randomized evaluation (loader configuration in Appendix E), all 20 evaluated judges produce κ values in the range [0.616, 0.898].The top five judges produce highly reasonable κ values: Gemini 3.1 Pro (0.898), GPT-oss 120B (0.880), Claude Opus 4.6 (0.879), GPT-5.4 (0.879), and Claude Haiku 4.5 (0.873).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Kappa deflation is universal across the 21 evaluated models on MT-Bench, with exact match overstating chance-corrected agreement by 33.8 to 41.3 percentage points (cohort mean 38.6 pp).

    “kappa deflation between exact match and Cohen's κ is universal (33-41 pp on MT-Bench)”
    • Also stated in: § Experimental Procedure · plain
      “Every judge in our study exhibits substantial kappa deflation on MT-Bench: exact match overstates chance-corrected agreement by between 33.8 and 41.3 percentage points across the 21 models, with a cohort mean of 38.6 pp.”
    • Also stated in: § 1 Introduction · plain
      “First, kappa deflation: raw agreement overstates chance-corrected discrimination by 33-41pp in all 21 evaluated models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All 21 judges evaluated under the bias-audit protocol register a verbosity bias below 0.011 on MT-Bench, with the largest value being GPT-4o-mini at 0.010; 17 of the 21 judges fall below 0.005.

    “All 21 judges evaluated under the bias-audit protocol register a verbosity bias below 0.011 on MT-Bench, with the largest value being GPT-4o-mini at 0.010; 17 of the 21 judges fall below 0.005, with the smallest values clustered around 0.001.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Kimi K2.5 records the lowest position bias of any non-Gemini judge (0.004), competitive with Claude Opus 4.6, while achieving JudgeBench κ = 0.720 at a fraction of the frontier-tier cost.

    “Mid-tier judges can outperform frontier judges on specific dimensions: Kimi K2.5 records the lowest position bias of any non-Gemini judge (0.004), competitive with Claude Opus 4.6, while achieving JudgeBench κ = 0.720 at a fraction of the frontier-tier cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among the three thinking-architecture judges (GPT-5.4, Gemini 3.1 Pro, DeepSeek V3.2), only Gemini 3.1 Pro (0.038) has position bias below 0.05, contrary to H2.

    “H2 predicted that the three thinking-architecture judges (GPT-5.4,Gemini 3.1 Pro, DeepSeek V3.2) would all fall below 0.05; only Gemini 3.1 Pro (0.038) does so.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    H3: Cross-benchmark rank divergence: 11 of 21 judges with maximum pairwise rank shift ≥4; max shift 15 (Llama 3.3 70B). Confirmed.

    “H3 Cross-benchmark rank diver- | Some model exhibits | 11 of 21 judges with maximum pair- | Confirmed | §4.3”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    H5: RewardBench remains degenerate under generative evaluation: κ ≤0.05 across all judges; observed κ ∈ [0.616, 0.898] across 21 judges. Refuted.

    “H5 RewardBench remains de- | κ ≤ 0.05 across all | κ ∈ [0.616, 0.898] across 21 | Refuted | §4.5”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The RewardBench item count is 2,981, verified from post-randomization agreement runs.

    “The RewardBench item count of 2,981 is the n_samples value verified from the post-randomization agreement runs”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Approximately 541,000 judgments were produced.

    “Approximately 541,000 judgments were produced.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The released artifact is an evaluation dataset of 118 evaluation runs spanning 21 LLM judges, three benchmarks, and three protocols (approximately 541,000 individual judgments).

    “The released artifact is an evaluation dataset of 118 evaluation runs spanning 21 LLM judges, three benchmarks, and three protocols (approximately 541,000 individual judgments).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    Seven of the sixteen judges evaluated under the consistency protocol show a position flip-rate increase of at least 1.5× from MT-Bench to JudgeBench; the mean test-retest across the 16 judges drops from 0.943 on MT-Bench to 0.911 on JudgeBench.

    “Consistent with Hypothesis 6, seven of the sixteen judges evaluated under the consistency protocol show a position flip-rate increase of at least 1.5× from MT-Bench to JudgeBench.”
    • Also stated in: § 4.6 Consistency Falls on Hard Benchmarks · plain
      “Test-retest reliability follows a similar pattern at the cohort level: the mean test-retest across the 16 judges drops from 0.943 on MT-Bench to 0.911 on JudgeBench.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MT-Bench compresses the 21 judges into a 13.5 pp κ band (0.376 to 0.511), while JudgeBench spreads them over 60.4 pp (0.271 to 0.875), a 4.5× wider range; RewardBench falls between at 28.1 pp (0.616 to 0.898).

    “MT-Bench compresses all 21 judges into a 13.5 pp κ band (0.376 to 0.511) with an average gap of 0.6 pp between adjacent ranks; JudgeBench spreads the same models over 60.4 pp (0.271 to 0.875) with 3-9 pp tiers; RewardBench falls between at 28.1 pp (0.616 to 0.898), with the top seven judges within a tight 2.7 pp band followed by gradual separation.”
    • Also stated in: § 4.4 MT-Bench Ceiling Effect · plain
      “JudgeBench's κ spread on the same population is 60.4 pp, a factor of 4.5× wider”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Position bias varies widely across judges: least biased Gemini 2.5 Pro (pb=0.002) and most biased Qwen 3 8B (pb=0.192) span nearly two orders of magnitude; reasoning and frontier models have systematically lower position bias than small, cost-optimized models.

    “The least position biased model, Gemini 2.5 Pro (pb = 0.002) and most position biased model Qwen 3 8B (pb = 0.192) span nearly two orders of magnitude difference in position bias.In this evaluation, reasoning and frontier models have systematically lower rates of position bias; small, and cost-optimized models have the highest rates of position bias.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

    Findingown data

    H2: Thinking-architecture judges (GPT-5.4, Gemini 3.1 Pro, DeepSeek V3.2) show low position bias: Gemini 3.1 Pro 0.038 (yes); GPT-5.4 0.083 (no); DeepSeek V3.2 0.094 (no). Partially confirmed (1 of 3).

    “H2: thinking architecture does not reliably reduce position bias”
    • Also stated in: § G Use of AI Assistance · plain
      “H2 Thinking-architecture judges | Position bias < 0.05 | Gemini 3.1 Pro 0.038 (yes); GPT- | Partially con- | §4.2”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    Cohort mean test-retest reliability on MT-Bench is 0.944, on JudgeBench 0.913; self-consistency 0.969 and 0.950; position flip rate 0.099 and 0.173.

    “Cohort mean | 0.944 | 0.969 | 0.099 | 0.913 | 0.950 | 0.173”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Findingown data

    Cohort mean Δκ on MT-Bench is 38.6 pp, on JudgeBench 24.0 pp, on RewardBench 10.4 pp.

    “Cohort mean (21) | | | | 38.6 | | | | 24.0 | | | 10.4”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

  • May combine two claims.

    Study design and conductown data

    The validity of the corrected loader is confirmed by the resulting κ distribution: across the 21 judges evaluated on RewardBench, κ values fall in the range [0.616, 0.898], well within the substantial-to-near-perfect agreement bands.

    “The validity of the corrected loader is confirmed by the resulting κ distribution: across the 21 judges evaluated on RewardBench, κ values fall in the range [0.616, 0.898], well within the substantial-tonear-perfect agreement bands.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    AI assistance was not used for original writing of research claims, code generation, research design, hypothesis formulation, data collection, or data analysis.

    “AI assistance was not used for original writing of research claims, for code generation in the evaluation experiments, or for any aspect of research design, hypothesis formulation, data collection, or data analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Verbosity sensitivity has fallen substantially over the past two model generations, as the observed verbosity bias magnitudes are an order of magnitude smaller than the variance contributions reported in 2023-era studies.

    “The magnitudes are an order of magnitude smaller than the variance contributions reported in 2023-era studies of style and length effects (Wu and Aji, 2025), suggesting that verbosity sensitivity has fallen substantially as a practical concern over the past two model generations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wu and Aji, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The study suppressed the reasoning channel for thinking models to maintain judge comparability and prevent verdict-token truncation; results do not characterize the thinking-on configuration.

    “For models that generate a built-in reasoning trace (GPT-5.4,GPT-5.4-mini,Gemini 3.1 Pro, DeepSeek V3.2, Kimi K2.5, GLM-5, Minimax M2.7), the reasoning channel was suppressed using the provider-specific mechanisms documented in Appendix D; this maintained judge comparability and prevented verdicttoken truncation when reasoning consumed the output budget.Reasoning-enabled evaluations could change”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study used temperature 0 for all judges.

    “Temperature is 0 for all judges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study used position-swap (AB+BA) debiasing in the consistency and bias-audit protocols.

    “position-swap (AB+BA) debiasing in the consistency and biasaudit protocols.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    Per-item randomization with a fixed seed restores a balanced ordering distribution and produces a valid agreement signal.

    “Per-item randomization with a fixed seed restores a balanced ordering distribution and produces a valid agreement signal.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All experimental results, metric computations, and findings were produced and verified by the authors using the framework and methods described in Sections 3 and 4.

    “All experimental results, metric computations, and findings reported in this paper were produced and verified by the authors using the framework and methods described in Sections 3 and 4.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The study's findings are limited to English-language, text-only benchmarks and should not be extrapolated to multilingual or multimodal judging without further measurement.

    “Our evaluation uses three established English-language, text-only benchmarks (MT-Bench, JudgeBench, RewardBench).Multilingual judging and multimodal judgingboth of which are increasingly common deployment settings-are not characterized in this study (Fu and Liu, 2025).We caution against extrapolating any of our findings to those settings without further measurement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fu and Liu, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    Total run count is 118, matching the production state ledger.

    “Total run count is 118, matching the production state ledger.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Test-retest reliability measures the stability of a judge's outputs, not the correctness of the underlying decision process; position bias and within-judge agreement are mathematically orthogonal.

    “The paradox arises because test-retest measures the stability of outputs and not the correctness of the underlying decision process: a judge that deterministically favors position A across runs would achieve a perfect test-retest score, but would also exhibit maximum-possible position bias.”
    • Also stated in: § 5.3 The Consistency-Bias Paradox and a Minimum Viable Validation Protocol · plain
      “Test-retest reliability measures output stability, not decision-process correctness.Position bias and within-judge agreement are mathematically orthogonal, so a judge that deterministically prefers position A across runs achieves near-perfect test-retest while exhibiting maximal position bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    LLM-as-a-Judge has become the dominant evaluation paradigm for language models and is widely deployed in production evaluation pipelines.

    “LLM-as-a-Judge has become the dominant evaluation paradigm for language models”
    • Also stated in: § 1 Introduction · plain
      “Such judges are widely deployed in production evaluation pipelines across hundreds of documented LLMOps deployments (DoorDash Engineering, 2024;Strick van Linschoten, 2025).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites DoorDash Engineering, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Strick van Linschoten, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The study's results are a snapshot of provider behavior during March-April 2026 and may not reflect temporal stability due to silent endpoint drift.

    “All evaluation runs were executed within a five-week window in March-April 2026.Hosted-model endpoints are known to drift across provider-side updates, sometimes silently, and we have not yet re-evaluated these judges over time to quantify within-judge stability over a longer time horizon.Consequently, the numbers we report should be interpreted as a snapshot of provider behavior during this speci”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Hosted-model endpoints drift across provider-side updates, sometimes silently.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Position-swap debiasing (AB+BA paired evaluations) was applied in both the consistency and bias-audit protocols.

    “Position-swap debiasing (AB+BA paired evaluations) was applied in both the consistency and bias-audit protocols.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The predictions were recorded internally and were not deposited with an external registry; the paper does not claim formal pre-registration.

    “The predictions were recorded internally and were not deposited with an external registry; we do not claim the methodological status of formal pre-registration.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    A model's apparent ranking on a single benchmark is a poor estimator of its ranking on others because benchmarks differ in discriminative power and measure subtly different latent constructs.

    “Where the underlying distribution is compressed, small κ differences produce large rank changes, and a model's apparent ranking on a single benchmark is a poor estimator of its ranking on others.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The RewardBench evaluation uses per-item position randomization with seed 42, matching the official run_generative.py script.

    “The RewardBench evaluation in this study uses per-item position randomization with seed 42, matching the contract of the official run_generative.pyscript that ships with the benchmark.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Seven hypotheses (H1-H7) were formulated on 2026-04-14, before Phase 6 data collection began, based on patterns observed in earlier phases (Phases 1-5).

    “Seven hypotheses (H1-H7) were formulated on 2026-04-14, before Phase 6 (frontier-model) data collection began, based on patterns observed in the earlier phases (Phases 1-5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The evaluation campaign was conducted in seven phases over a five-week window in March and April 2026.

    “The evaluation campaign was conducted in seven phases over a five-week window in March and April 2026.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The consistency protocol was executed with response caching disabled to ensure replicate runs sample independent generations rather than memorized responses.

    “In order to ensure that replicate runs sample independent generations rather than memorized responses, we execute the consistency protocol with response caching disabled.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    AI assistance was used only for editing and manuscript-formatting support.

    “The authors used AI assistance (large language model-based writing assistants) for editing and manuscript-formatting support only: specifically, copy-editing prose, normalizing table and reference formatting, debugging LaTeX layout, and routine BibTeX hygiene.”
    Quote in the paperNot a substantial claim
50 of 50

  • The paper relies on this without support.

    No prior work has conducted a large-scale, cross-benchmark, multi-protocol evaluation of judge reliability across model families and generations.

    “No prior work has conducted a large-scale, crossbenchmark, multi-protocol evaluation of judge reliability across model families and generations.”

  • The paper relies on this without support.

    Hosted-model endpoints drift across provider-side updates, sometimes silently.

    “Hosted-model endpoints are known to drift across provider-side updates, sometimes silently, and we have not yet re-evaluated these judges over time to quantify within-judge stability over a longer time horizon.”

  • The paper relies on this without support.

    Rubric design choices, including reference grounding and reasoning instructions, interact with bias profiles in ways the single-template design cannot isolate.

    “Rubric design choices, including reference grounding and reasoning instructions, are known to interact with bias profiles in ways our single-template design cannot isolate.”

  • The paper relies on this without support.

    Token-level logprobs are not exposed by the majority of providers in the cohort.

    “Expected Calibration Error and Brier Score require token-level logprobs that are not exposed by the majority of providers in our cohort.”

  • The paper relies on this without support.

    Hypotheses formulated post hoc would typically be confirmed at a higher rate than a priori hypotheses.

    “hypotheses formulated post hoc would typically be confirmed at a higher rate.”

  • The paper relies on this without support.

    The hosted-model endpoints did not drift during the evaluation window (March-April 2026).

    “interpreting the released evaluation dataset as authoritative beyond its temporal scope (March-April 2026) for judges whose hosted-model endpoints subsequently drift”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of judges with large rank shifts

    Minor

    The discussion says ten judges have a maximum pairwise rank shift of at least four positions, while the hypothesis-outcome table says eleven.

  • RewardBench judge count is reported as both 20 and 21

    Minor

    Section 4.5 says 20 judges were evaluated on RewardBench. Appendix E and the per-judge results report 21.

  • Figure understates the maximum MT-Bench kappa deflation

    Minor

    The figure caption gives a maximum deflation of 41.2 percentage points, but Table 2 reports 41.3 for GPT-4o-mini.

  • Cohort-mean deflation differs between tables

    Minor

    Table 4 gives JudgeBench and RewardBench cohort means of 24.0 and 10.4 percentage points, while Table 2 gives 23.7 and 10.2 for the same 21 judges.

  • Phase-listed consistency runs do not total 34

    Minor

    The phase table lists 12 consistency runs in Phase 4 and 20 in Phase 7; no other phase lists the consistency protocol. Those phase counts total 32, rather than the stated 34 consistency runs.

    Reported 34 · recomputed 32

  • RewardBench's stated top five omits the second-ranked judge

    Minor

    The stated top-five list omits Claude Sonnet 4, which Table 5 ranks second on RewardBench at κ = 0.886. It includes Claude Haiku 4.5, ranked sixth.

  • Minimax's stated MT-Bench rank differs from the rank table

    Minor

    The discussion places Minimax M2.7 at MT-Bench rank 16, whereas the cross-benchmark rank table places it at rank 17. Both give its JudgeBench rank as 5.

  • Consistency cohort is reported as both 16 and 17 judges

    Minor

    The results describe a 16-judge consistency cohort, but the appendix explicitly identifies 17 judges in that cohort and lists 17 per-judge rows.

  • Llama's cross-benchmark rank shift is 15, not 14

    Minor

    The rank table puts Llama 3.3 70B at MT-Bench rank 5 and JudgeBench rank 20, a 15-position shift. The figure caption instead gives JudgeBench rank 19 and a 14-position shift; the paper's headline maximum of 14 is therefore inconsistent with its table.

9 of 9

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 17

  1. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301-8327, Miami, Florida, USA. Association for Computational Linguistics. · DOI · Paper page
  2. Xiyan Fu and Wei Liu. 2025. How reliable is multilin- gual LLM-as-a-judge? In Findings of the Associa- tion for Computational Linguistics: EMNLP 2025, pages 11040-11053, Suzhou, China. Association for Computational Linguistics. · DOI
  3. Rajarshi Haldar and Julia Hockenmaier. 2025. Rat- ing roulette: Self-inconsistency in LLM-as-a-judge frameworks. In Findings of the Association for Com- putational Linguistics: EMNLP 2025, pages 24986- 25004, Suzhou, China. Association for Computa- tional Linguistics. · DOI · Paper page
  4. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024a. Prometheus: Inducing fine- grained evaluation capability in language models. In International Conference on Learning Representa- tions (ICLR).
  5. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. Re- wardBench: Evaluating reward models for language modeling. In Findings of the Association for Compu- tational Linguistics: NAACL 2025, pages 1755-1797, Albuquerque, New Mexico. Association for Compu- tational Linguistics. · DOI · Paper page
  6. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat- tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025a. From gen- eration to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 2757-2791, Suzhou, China. Asso- ciation for Computational Linguistics. · DOI
  7. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2025b. From crowdsourced data to high- quality benchmarks: Arena-hard and benchbuilder pipeline. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net.
  8. Zihao Li, Feihao Fang, Xitong Zhang, Jiaru Zou, Zhin- ing Liu, Wei Xiong, Ziwei Wu, Baoyu Jing, and Jin- grui He. 2025c. Not all voices are rewarded equally: Probing and repairing reward models across human diversity. In Findings of the Association for Com- putational Linguistics: EMNLP 2025, pages 3426- 3455, Suzhou, China. Association for Computational Linguistics. · DOI
  9. Zongjie Li, Chaozheng Wang, Pingchuan Liu, Daoyuan Wang, Dong Yang, Shuai Wang, and Cuiyun Liu. 2024b. Split and merge: Aligning position biases in LLM-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP).
  10. Bill Yuchen Lin, Yuntian Deng, Khyathi Raghavi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2025. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. In The Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net.
  11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human align- ment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511-2522, Singapore. Association for Com- putational Linguistics. · DOI · Paper page
  12. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. 2026. RewardBench 2: Advancing reward model evaluation. In International Confer- ence on Learning Representations (ICLR).
  13. Sijun Tan, Sana Mavandadi, Amir Tan, Rui Tan, Dong- Ho Tan, and Arash Mahyari. 2025. JudgeBench: A benchmark for evaluating LLM-based judges. In In- ternational Conference on Learning Representations (ICLR).
  14. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large lan- guage models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9450, Bangkok, Thailand. Association for Computational Linguistics. · DOI · Paper page
  15. Victor Wang, Michael JQ Zhang, and Eunsol Choi. 2025. Improving LLM-as-a-judge inference with the judg- ment distribution. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23173-23199, Suzhou, China. Association for Com- putational Linguistics. · DOI
  16. Minghao Wu and Alham Fikri Aji. 2025. Style over sub- stance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 297-312, Abu Dhabi, UAE. Association for Computational Linguis- tics.
  17. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. JudgeLM: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations (ICLR).

Cited by

Sorted by citations

Related

Authors
Norman, J. D., Rivera, M. U., & Hughes, D. A.
Affiliations
Berkeley College (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.19544
arXiv
2606.19544
OpenAlex
W7165368332
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
LLM-as-a-judge, language model evaluation, Cohen's kappa, test-retest reliability, position bias, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-03.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-03.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review