CoChat
Sign Up
PreprintCited by 0Topic ModelingNatural Language Processing TechniquesMulti-Agent Systems and Negotiation

Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

Open access
Song, Minsoo, Kim, Chanwoo, Eo, Sugyeong, Park, Chanjun·2026·arXiv
DOI 10.48550/arxiv.2608.30373arXiv 2608.30373
Open this paper in CoChat — free accountOpen PDF

Abstract

Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

AI snapshot

Written from the paper’s full text.

TL;DR

In subjective rubric-based evaluation, a single LLM judge aligned better with human reference scores than consensus-based multi-agent debate (MAD) across six judge models and two tasks. The degradation is driven mainly by asymmetric strict-lenient role prompting, which produces a systematic downward score bias that consensus fails to correct.

Key findings

  • The Single Judge baseline achieved the strongest average human alignment: Korean Essay RMSE 0.644, ρ 0.504; SummEval RMSE 0.682, ρ 0.458. Consensus-based MAD degraded both: Korean Essay RMSE 0.935, ρ 0.424; SummEval RMSE 0.813, ρ 0.409.
  • The Strict Role Judge alone shifted scores downward and worsened RMSE: mean scores 2.733 vs Single Judge 3.345 and human reference 3.486 on Korean Essay; 3.672 vs 3.935 and 4.285 on SummEval. Its RMSE was 0.973 on Korean Essay and 0.884 on SummEval, higher than Consensus-based MAD’s 0.935 and 0.813.
  • Symmetric MAD, which removes role asymmetry while preserving multi-round interaction, largely recovered baseline performance: RMSE 0.733 on Korean Essay and 0.687 on SummEval, close to Single Judge’s 0.644 and 0.682; mean scores 3.249 and 3.964 vs Single Judge’s 3.345 and 3.935.
  • Score masking worsened average RMSE and widened the final inter-agent gap: Korean Essay RMSE 0.935 → 1.040, gap 0.176 → 0.385; SummEval RMSE 0.813 → 0.994, gap 0.076 → 0.224. Consensus shifts (−0.589 Korean Essay, −0.454 SummEval) fell beyond the arithmetic midpoint of Strict-only and Lenient-only shifts (−0.334 and −0.335), indicating strict-stance dominance beyond averaging.

Methods

  • Design: compared a Single Judge baseline against Consensus-based MAD with Strict and Lenient agents interacting up to Round 4 and final scores averaged arithmetically, plus three ablations: Strict Role Judge, Symmetric MAD, and Score-Masked MAD. Task input, rubric, scoring scale, and judge model were fixed across protocols.
  • Data: Korean Essay Scoring used the NIKL Grading Writing Data 2024, sampling 600 essays from six topics, scored on content, organization, and expression, with the mean of two human evaluators as reference. SummEval used 700 system-summary instances (100 articles × 7 systems), scored on coherence, consistency, fluency, and relevance, with the mean of three annotators as reference.
  • Models and analysis: six LLMs (GPT-4o-mini, Gemma-3-4B/12B/27B, Qwen3.5-9B/27B) at temperature 0 via OpenRouter; metrics were RMSE and Spearman correlation, averaged over six judge models, with paired-bootstrap significance tests reported.

Limitations

  • Stated: the MAD protocol is a controlled operationalization rather than a direct replication; only one role configuration (strict-lenient) is tested; the protocol is fixed to five total rounds; final scores use unweighted averaging; findings cover two domains and six judge models; several design axes remain unexplored, including number of agents, role diversity, interaction format, aggregation rule, and judge specialization.
  • Stated: the SummEval single-judge prompt includes “strict and consistent evaluator,” a directional cue absent from the Korean Essay baseline; this asymmetry may affect cross-domain comparisons and is not directly quantified.
  • Evident: the aggregate ordering is dominant but not universal—the paper reports Qwen3.5-27B Symmetric MAD outperforming Single Judge on both tasks (Korean Essay RMSE 0.864 vs 0.938; SummEval 0.734 vs 0.799) and Gemma3-4B showing small differences across protocols—so the conclusions reflect a central tendency rather than a universal rule.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2608.30373

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 16 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § Abstract · p. 1

    Removing role asymmetry (Symmetric MAD) largely recovers baseline performance.

    “Removing role asymmetry (Symmetric MAD) largely recovers baseline performance”
    • Also stated in: § 4.2 Role Asymmetry · p. 4 · plain
      “By contrast, Symmetric MAD remains much closer to the single-judge baseline.”
    • Also stated in: § 5 Conclusion · p. 5 · plain
      “Symmetric MAD, which retains multi-round interaction without role asymmetry, shows improved alignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.
  • Findingown data
    § Abstract · p. 1

    The consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out.

    “The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.
  • Findingown data
    § 4.3 Cross-Model Generalization · p. 5

    Qwen3.5-9B S / Gemma3-27B L yields the largest errors on both tasks.

    “Qwen3.5-9B S /Gemma3-27B L yields the largest errors on both tasks”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.
  • Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Gemma3-27B's gap decreases gradually (0.47 to 0.08).

    “Gemma3-27B's gap decreases gradually (0.47 to 0.08)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.
  • Findingown data
    § D Score-Masked MAD analysis · p. 14

    In those same cases the inter-agent gap widens considerably.

    “but in those same cases the inter-agent gap widens considerably”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 34
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    LLMs can provide scalable and rubric-aware evaluations for summarization, dialogue, and long-form responses, often showing stronger correlations with human judgments than traditional metrics.

    “Prior studies have shown that LLMs can provide scalable and rubric-aware evaluations for summarization, dialogue, and long-form responses, often showing stronger correlations with human judgments than traditional metrics (Liu et al., 2023; Zhu et al., 2025; Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Liu et al., 2023 source

      The Introduction and Conclusion describe G-EVAL as an evaluation method guided by specified criteria and report stronger correlations with human judgments for summarization and dialogue generation than comparison metrics. The paper does not report evaluations of long-form responses or establish that the method is scalable; those parts of the citing sentence are not supported by this source. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Zhu et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Kim et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 Introduction · p. 1

    LLM judges are known to exhibit systematic biases, such as self-preference, position bias, and unfair evaluation behavior.

    “However, LLM judges are also known to exhibit systematic biases, such as self-preference, position bias, and unfair evaluation behavior (Wataoka et al., 2024; Shi et al., 2025; Wang et al., 2024).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Wataoka et al., 2024 source
    • cites Shi et al., 2025 source
    • cites Wang et al., 2024 source

      Section 2.2 reports that GPT-4 and ChatGPT change their evaluations when candidate-response positions are swapped, and Section 2.3 says this positional bias can make evaluations unfair. The paper does not establish self-preference—favoring a judge’s own outputs—so that example is not supported by this source. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    MAD has been proposed as a way to improve the reasoning and factuality of LLM outputs by allowing multiple agents to exchange and revise their answers.

    “In parallel, Multi-Agent Debate (MAD) has been proposed as a way to improve the reasoning and factuality of LLM outputs by allowing multiple agents to exchange and revise their answers (Du et al., 2024; Liang et al., 2024).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Du et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liang et al., 2024 source

      The Introduction says MAD lets agents correct one another and achieve more accurate answers, and Section 2 describes debaters exchanging arguments across iterations before a judge selects a solution. The paper reports improvements on translation and arithmetic-reasoning tasks, but does not establish improved factuality of LLM outputs more broadly. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    MAD can be effective in reasoning-oriented tasks where answers are relatively verifiable, such as factual question answering or mathematical reasoning.

    “Such applications are supported by evidence that MAD can be effective in reasoning-oriented tasks where answers are relatively verifiable, such as factual question answering or mathematical reasoning (Liang et al., 2024; Zhang and Xiong, 2025).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Liang et al., 2024 source

      Section 3.4 reports that MAD outperformed the other GPT-3.5-Turbo-based methods on counter-intuitive arithmetic reasoning and says the authors also validated it on math and symbolic reasoning tasks. That supports the mathematical-reasoning example, but the paper’s stated experiments are arithmetic reasoning and commonsense machine translation, not factual question answering; that example is not established by this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Zhang and Xiong, 2025 source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 Introduction · p. 1

    Recent work has extended MAD to LLM-based evaluation, introducing multi-agent or debate-based judges for open-ended text evaluation, evaluator meta-evaluation, and machine translation evaluation.

    “Recent work has extended this idea to LLM-based evaluation, introducing multi-agent or debate-based judges for openended text evaluation, evaluator meta-evaluation, and machine translation evaluation (Chan et al., 2024; Li et al., 2024; Chern et al., 2024; Feng et al., 2025; Chen et al., 2025).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Chan et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Li et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chern et al., 2024 source

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    The single-judge baseline achieves the strongest human alignment on average across six judge models on both tasks, in both RMSE and Spearman correlation.

    “the single-judge baseline achieves the strongest human alignment on average across six judge models”
    • Also stated in: § 5 Conclusion · p. 5 · plain
      “Across both tasks, the single-judge baseline outperformed all multi-agent conditions in terms of RMSE and Spearman correlation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    Consensus-based MAD degrades human alignment relative to the single-judge baseline on both tasks, increasing RMSE to 0.935 (Korean Essay) and 0.813 (SummEval) and decreasing Spearman correlation to 0.424 and 0.409.

    “MAD shows degradation in human alignment on both tasks”
    • Also stated in: § 4.1 Human Alignment Across Protocols · p. 4 · hedged
      “Consensus-based MAD increases RMSE to 0.935 and 0.813 while Spearman drops to 0.424 and 0.409, indicating degradation in both absolute agreement and ranking preservation.”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “Our experiments show that consensus-style multi-agent judging degrades alignment with human reference scores”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    Masking peer scores widens inter-agent disagreement on average and worsens average human alignment.

    “masking peer scores widens inter-agent disagreement on average and worsens average human alignment”
    • Also stated in: § 5 Conclusion · p. 5 · plain
      “Score exposure acts more as a coordination signal than a harmful anchor, as masking scores widens the inter-agent gap without recovering human alignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § 4.1 Human Alignment Across Protocols · p. 4

    RMSE generally increases after the initial exchange, indicating that subsequent rounds fail to correct the degradation and instead compound it.

    “As illustrated in Figure 2, RMSE generally increases after the initial exchange. This indicates that subsequent rounds fail to correct the degradation and instead compound it.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Figure 3 shows the direct cause of RMSE stability for Gemma3-27B and Gemma3-4B: their mean predicted scores remain close to the gold mean throughout all rounds on both tasks.

    “Figure 3 shows the direct cause of RMSE stability for Gemma3-27B and Gemma3-4B: their mean predicted scores remain close to the gold mean throughout all rounds on both tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E Model Specific Analysis · p. 15

    Gemma3-27B shows a better RMSE under Symmetric MAD on Korean Essay (0.512 vs. 0.528), although Single Judge retains an advantage on Spearman correlation and on SummEval.

    “Gemma3-27B also shows a better RMSE under Symmetric MAD on Korean Essay (0.512 vs. 0.528), although Single Judge retains an advantage on Spearman correlation and on SummEval.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § 4.3 Cross-Model Generalization · p. 5

    Cross-model pairing changes the magnitude of RMSE but does not eliminate the role-asymmetry pattern; no configuration outperforms the strongest single-judge baseline.

    “cross-model pairing changes the magnitude of RMSE but does not eliminate the role-asymmetry pattern: no configuration outperforms the strongest single-judge baseline”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Qwen3.5-27B starts at 2.76 (already 0.73 below gold) and falls further to 2.32, and GPT-4o-mini drops from 3.08 to 2.45 across rounds.

    “By contrast, Qwen3.5-27B starts at 2.76 (already 0.73 below gold) and falls further to 2.32, and GPT-4o-mini drops from 3.08 to 2.45 across rounds.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E Model Specific Analysis · p. 15

    For Qwen3.5-27B, Symmetric MAD outperforms the Single Judge on both tasks (Essay: 0.864 vs. 0.938 RMSE; SummEval: 0.734 vs. 0.799 RMSE).

    “For Qwen3.5-27B, Symmetric MAD outperforms the Single Judge on both tasks (Essay: 0.864 vs. 0.938 RMSE; SummEval: 0.734 vs. 0.799 RMSE)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § 4.2 Role Asymmetry · p. 4

    Interaction in Consensus-based MAD partially moderates the strict-role shift, but does not restore single-judge alignment.

    “Interaction in Consensus-based MAD partially moderates this strict-role shift, but does not restore single-judge alignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § 4.2 Role Asymmetry · p. 4

    The Strict Role Judge condition yields higher RMSE than Consensus-based MAD on both tasks (Essay: 0.973 vs. 0.935; SummEval: 0.884 vs. 0.813).

    “The same condition also yields higher RMSE than Consensus-based MAD (Essay: 0.973 vs. 0.935; SummEval: 0.884 vs. 0.813), indicating that the strict role prompt alone moves predictions away from the human reference.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 4.2 Role Asymmetry · p. 4

    Consensus-based MAD produces lower mean scores than both the single-judge baseline and the human reference on both tasks: on Korean Essay, 2.900 vs. 3.345 and 3.486; on SummEval, 3.831 vs. 3.935 and 4.285.

    “Consensus-based MAD produces lower mean scores than both the single-judge baseline and the human reference on both tasks.”
    • Also stated in: § 4.2 Role Asymmetry · p. 4 · plain
      “On Korean Essay, the mean score is 2.900 under Consensus-based MAD, compared with 3.345 under Single Judge and 3.486 for the human reference.”
    • Also stated in: § 4.2 Role Asymmetry · p. 4 · plain
      “On SummEval, the corresponding values are 3.831, 3.935, and 4.285.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Findingown data
    § E Model Specific Analysis · p. 15

    Gemma3-12B shows a similar vulnerability to score masking on Korean Essay, where Score-Masked MAD reduces Spearman to 0.173-the lowest value observed across all model-protocol combinations.

    “Gemma3-12B shows a similar vulnerability to score masking on Korean Essay, where Score-Masked MAD reduces Spearman to 0.173-the lowest value observed across all modelprotocol combinations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Gemma3-12B's gap trajectory resembles Gemma3-27B's (Figure 4), yet its mean score drifts much more.

    “Notably, Gemma3-12B's gap trajectory resembles Gemma3-27B's (Figure 4), yet its mean score drifts much more.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingreasoning
    § B.1 Korean Essay Scoring · p. 13

    The final dataset for Korean Essay consists of 600 essays (100 essays per prompt × 6 prompts).

    “yielding a final dataset of 600 essays .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G.1 Aggregate Results · p. 16

    The lenient prompt does not produce an upward bias symmetric to the downward shift induced by the strict prompt: the Lenient-only shifts are +0.086 and -0.057, compared with -0.754 and -0.613 for Strict-only on Korean Essay and SummEval, respectively.

    “The lenient prompt does not produce an upward bias symmetric to the downward shift induced by the strict prompt: the Lenient-only shifts are +0 . 086 and -0 . 057 , compared with -0 . 754 and -0 . 613 for Strict-only on Korean Essay and SummEval, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § B.2 SummEval · p. 14

    The final dataset consists of 700 (article, summary) pairs (100 articles × 7 summarization systems).

    “The final dataset consists of 700 (article, summary) pairs (100 articles × 7 summarization systems)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Mean Predicted Score by Round (Consensus-based MAD, 6 models) · p. 17

    Lenient-only reduces RMSE on both tasks (Korean Essay: 0.644 → 0.582; SummEval: 0.682 → 0.498), but also lowers Spearman correlation (Korean Essay: 0.504 → 0.478; SummEval: 0.458 → 0.417).

    “Lenient-only reduces RMSE on both tasks (Korean Essay: 0 . 644 → 0 . 582 ; SummEval: 0 . 682 → 0 . 498 ), but also lowers Spearman correlation (Korean Essay: 0 . 504 → 0 . 478 ; SummEval: 0 . 458 → 0 . 417 ).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Neither Gemma3-27B nor Gemma3-4B's mean drifts far from the gold reference.

    “Neither model's mean drifts far from the gold reference.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    On Korean Essay (gold mean = 3.49), Gemma3-27B fluctuates between 3.50 and 3.57 across rounds, while Gemma3-4B rises only slightly from 3.50 to 3.61.

    “On Korean Essay (gold mean = 3 . 49 ), Gemma3-27B fluctuates between 3.50 and 3.57 across rounds, while Gemma3-4B rises only slightly from 3.50 to 3.61.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Findingown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Gemma3-12B's mean starts below gold (3.06) and continues declining to 2.79, consistent with its moderate RMSE increase in Table 6.

    “Gemma3-12B occupies an intermediate position: its mean starts below gold (3.06) and continues declining to 2.79, consistent with its moderate RMSE increase in Table 6.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    Multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

    “These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    Assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct.

    “assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct”
    • Also stated in: § 4.2 Role Asymmetry · p. 4 · plain
      “This downward shift is most pronounced in the Strict Role Judge condition.”
    • Also stated in: § 5 Conclusion · p. 5 · plain
      “The strict role prompt alone shifts scores substantially below the human reference distribution”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.
    • Rests on an assumption: Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

  • May combine two claims.

    Interpretationreasoning
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    Because neither agent's scoring stance dominates the deliberation, the averaged score is not pulled systematically in any direction, keeping the mean near gold.

    “Because neither agent's scoring stance dominates the deliberation, the averaged score is not pulled systematically in any direction, keeping the mean near gold.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • May combine two claims.

    Study design and conductreasoning
    § 3.1 Tasks and Datasets · p. 3

    The direction of the SummEval prompt asymmetry is considered conservative with respect to the conclusions, although its magnitude is not directly quantified.

    “We therefore consider the direction of this asymmetry conservative with respect to our conclusions, although we do not directly quantify its magnitude.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § E.1 Round-wise Mean Score and Gap Trajectories · p. 15

    A converging gap does not guarantee mean score stability: what matters is whether the convergence point is close to the gold reference.

    “This dissociation shows that a converging gap does not guarantee mean score stability: what matters is whether the convergence point is close to the gold reference.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § F Significance Tests · p. 16

    The nonsignificant near-ties and reversals in the final five comparisons provide transparent counterexamples while showing that the aggregate finding is not driven by a small number of outliers.

    “The nonsignificant near-ties and reversals in the final five comparisons provide transparent counterexamples while showing that the aggregate finding is not driven by a small number of outliers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Limitations · p. 5

    The consensus-style MAD protocol studied is the authors' own controlled operationalization rather than a direct replication of any single prior system.

    “The consensus-style MAD protocol we study is our own controlled operationalization rather than a direct replication of any single prior system”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § D Score-Masked MAD analysis · p. 14

    Removing numerical scores reduces inter-agent convergence without restoring alignment with human ratings.

    “Removing them reduces inter-agent convergence without restoring alignment with human ratings”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Limitations · p. 5

    Final scores are aggregated by unweighted averaging; alternative strategies such as weighted ensembles or majority voting are not explored.

    “Final scores are aggregated by unweighted averaging; alternative strategies such as weighted ensembles or majority voting are not explored.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.1 Tasks and Datasets · p. 3

    Both tasks use a 1-5 scoring scale, criterion-level human reference scores, and an identical grid of judge protocols and judge models.

    “both tasks use a 1-5 scoring scale, criterion-level human reference scores, and an identical grid of judge protocols and judge models.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 2 Methodology · p. 2

    Across all protocols, the task input, target text, rubric, scoring scale, and judge model are kept fixed.

    “Across all protocols, we keep the task input, target text, rubric, scoring scale, and judge model fixed.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § B.2 SummEval · p. 14

    SummEval provides human quality annotations for machine-generated summaries of CNN/DailyMail news articles, covering four dimensions: coherence, consistency, fluency, and relevance, each rated on a 1-5 scale by three independent human annotators.

    “SummEval provides human quality annotations for machine-generated summaries of CNN/DailyMail news articles, covering four dimensions: coherence , consistency , fluency , and relevance , each rated on a 1-5 scale by three independent human annotators.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites SummEval

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § B.1 Korean Essay Scoring · p. 13

    The original scoring rubric covers three dimensions: content (maximum 25 points), organization (maximum 10 points), and expression (maximum 10 points).

    “The original scoring rubric covers three dimensions: content (maximum 25 points), organization (maximum 10 points), and expression (maximum 10 points).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites NIKL Grading Writing Data 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Models and Metrics · p. 3

    Within each run, the same model performs all judge roles.

    “Within each run, the same model performs all judge roles.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § B.1 Korean Essay Scoring · p. 13

    The NIKL Grading Writing Data 2024 corpus provides structured JSON documents in which each essay is recorded at the paragraph level, accompanied by rubric-based holistic scores assigned independently by two trained human annotators.

    “The corpus provides structured JSON documents in which each essay is recorded at the paragraph level, accompanied by rubric-based holistic scores assigned independently by two trained human annotators.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites NIKL Grading Writing Data 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S1 (asserted), S2 (checkable), S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § Limitations · p. 5

    The SummEval single-judge prompt asymmetry may affect cross-domain comparisons and should be examined in future work.

    “this asymmetry may affect cross-domain comparisons and should be examined in future work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § Limitations · p. 5

    The study tests only one role configuration (strict-lenient) and fixes the protocol to five total rounds (initial plus four adjustment rounds).

    “we test only one role configuration (strict-lenient) and fix the protocol to five total rounds, consisting of an initial round followed by four adjustment rounds”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § Limitations · p. 5

    The study covers two domains (Korean essay scoring and English summarization) with six judge models; generalization to other tasks, languages, or model families requires further investigation.

    “Our study covers two domains (Korean essay scoring and English summarization) with six judge models; generalization to other tasks, languages, or model families requires further investigation.”
    Quote in the paperNot a substantial claim
44 of 44

  • The paper relies on this without support.

    Subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.

    “subjective scoring often involves a tension between penalizing weaknesses and recognizing plausible strengths.”

  • The paper relies on this without support.

    Human reference scores (average of two human evaluators for Korean Essay, human ratings for SummEval) are valid ground truth for measuring judge-to-human alignment.

    “The average of two human evaluators is leveraged as the reference score.”
    “the human rating is used as the reference score.”

  • The paper relies on this without support.

    The arithmetic mean of human annotator scores is a valid gold reference for subjective evaluation.

    “the mean serves as the gold reference throughout all experiments.”
    “the gold score per criterion is computed as the arithmetic mean of the three annotators' scores.”

  • The paper relies on this without support.

    The sampled datasets (600 essays, 700 pairs) are representative of the broader tasks.

    “we randomly sample 100 essays without replacement (seed = 42 , using Python's random.sample ), yielding a final dataset of 600 essays .”
    “We sample 100 news articles from the test split (seed = 42 , using HuggingFace's shuffle().select() )”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Strict Role Judge does not have the largest downward shift on SummEval

    Minor

    The Strict Role Judge has a SummEval mean of 3.672, but Score-Masked MAD has a lower mean of 3.667. Relative to the same human reference, the masked condition therefore has the more pronounced downward shift.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Forty-first international conference on machine learning .
  2. Liu. 2025. M-mad: Multidimensional multi-agent debate for advanced machine translation evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7084-7107.
  3. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. Judgelm: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations , volume 2025, pages 51257-51296.

Related

Authors
Song, M., Kim, C., Eo, S., & Park, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2608.30373
arXiv
2608.30373
OpenAlex
W7204953855
Semantic Scholar
a0cadb0b7abe8ffd99c83b1dcd28dedd53f90a47
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
multi-agent debate, LLM-as-a-judge, subjective evaluation, rubric-based assessment, downward bias, role asymmetry, human alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review