CoChat
Sign Up
PreprintCited by 10Topic ModelingNatural Language Processing TechniquesEthics and Social Impacts of AI

Quantifying and Mitigating Self-Preference Bias of LLM Judges

Open access
Yang, Jinming, Hu, Zheng, Qiu, Chuxian, Deng, Zhenyu, Jiao, Xinshan, Zhou, Tao·2026·arXiv
DOI 10.48550/arxiv.2604.22891arXiv 2604.22891
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5\% on average.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a fully automated, gold-standard-free framework to quantify and mitigate self-preference bias (SPB) in LLM-as-a-Judge systems by constructing equal-quality response pairs and differencing self-preference from a third-party baseline. Across 20 mainstream LLMs, it finds that high capability does not guarantee low SPB, and a structured multi-dimensional evaluation strategy reduces SPB by 31.5% on average.

Key findings

  • SPB varies widely: LongCat-Flash-Chat shows the strongest positive bias (β = 0.307), DeepSeek-V3.2 follows (β = 0.226), and Claude-Sonnet-4.5 shows the strongest negative bias (β = -0.229); 8 models have positive SPB, 9 negative, and 3 near zero, with 17 of 20 reaching significance under the “Sig (≥2/3)” rule.
  • Capability is decoupled from evaluative objectivity: the taxonomy classifies 3 models as Objective Judges, 6 as Machiavellian Judges, 7 as Incompetent Randomizers, and 4 as Blindly Biased Judges.
  • Task type matters: Text Generation has the most widespread significant SPB (10 models, mean β = 0.17), Information Provision has 8 significant models (mean β = 0.21), and Explanation has 7 (mean β = 0.25); Qwen3-235B-A22B-Thinking-2507 reaches β = 0.540 on Explanation and β = 0.466 on Information Provision.
  • The structured multi-dimensional mitigation reduces SPB by 31.5% on average across four high-bias models, with LongCat-Flash-Chat showing a 69.9% reduction; CoT sometimes yields slightly higher SPB than baseline.

Methods

  • The framework uses two benchmark judges, GPT-5-Chat-Latest and Gemini-2.5-Pro, to score responses on five dimensions (Relevance, Accuracy, Depth, Logic, Clarity) on a 0.0–10.0 scale with 0.25 steps; equal-quality pairs are defined by |score difference| ≤ ε = 0.25.
  • SPB is quantified as β_i = ρ_i − ρ_null_i, where ρ_i is the rate at which a model firmly selects its own response across both presentation orders on equal-quality pairs, and ρ_null_i is the analogous rate for a pre-designated third-party target response; significance uses a one-sided Binomial test, a two-proportion Z-test, and bootstrap confidence intervals (1,000 iterations).
  • Discriminability π_i is measured on high-contrast pairs (score gap ≥ 2.5), with π_i ≥ 0.8 as the threshold; mitigation replaces pointwise scoring with pairwise per-dimension forced-choice across the five dimensions, followed by majority vote, and is applied to models with |β_i| > 0.08.

Limitations

  • The paper states that measurements are defined relative to benchmark judges rather than an absolute notion of objectivity; the 100-sample setting may underpower fine-grained conclusions for some task types; family effects are not fully disentangled from pure self-preference; and experiments are primarily English-centric.
  • Evident from the text: human validation was conducted on deliberately selected hard cases near the ε-boundary, yielding 79.8% agreement and Fleiss’ κ = 0.64, while benchmark-judge consistency was Spearman ρ = 0.658; task-specific analyses were restricted to Text Generation, Information Provision, and Explanation because other categories had ≤ 6% of the sample.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2604.22891

Cites

75 of 77 references verified1 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 24 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Conventional metrics based on deviation from gold standards cannot reliably tell whether a strong model's preference for its own output arises from higher quality or from self-identification.

    “When a strong model selects its own output, conventional metrics based on deviation from gold standards cannot reliably tell whether the preference arises from higher quality or from self-identification (Panickssery et al., 2024;Liu et al., 2024).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Panickssery et al., 2024 source

      In “Controlling for groundtruth generation quality” (page unavailable), the paper says a model may justifiably prefer its own output if it is higher quality, and that ground-truth quality annotations would be needed to assess disproportionate self-preference. It does not evaluate conventional deviation-from-gold-standard metrics or establish that those metrics cannot reliably distinguish quality from self-identification. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Liu et al., 2024 source

      Section 4.1 reports self-bias in both reference-based and reference-free LM evaluator scores, and Section 4.3 says inflated scores did not correlate with human evaluations. The discussion of traditional reference-based metrics separately says ROUGE and BLEU can fail to measure content quality. The paper does not establish that those conventional gold-standard metrics specifically cannot distinguish a strong model’s higher-quality output from self-identification. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Current evaluation approaches conflate model capability with Self-Preference Bias (SPB).

    “Second, they conflate capability with Self-Preference Bias (SPB).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Panickssery et al., 2024 source
    • cites Liu et al., 2024 source
  • Findingown data

    The aggregated standard error of β_i is approximately σβ ≈ 0.022.

    “Based on the experimental data, the aggregated standard error of β i is be approximately σ β ≈ 0.022.”
    Quote in the paperSubstantial claim
  • Findingown data

    Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB.

    “Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB.”
    Quote in the paperSubstantial claim
  • Findingown data

    The Spearman rank correlation coefficient between the scores assigned by the two benchmark judges over all 2,000 scoring instances is ρ = 0.658.

    “The results yield ρ = 0.658, demonstrating statistically significant consistency between the two judges and confirming their shared capability to reliably distinguish high-quality from low-quality responses.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 21
  • May combine two claims. 33
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The 100-task sample was drawn from the AlpacaEval evaluation set using stratified sampling with a fixed random seed.

    “We sample 100 single-turn tasks from the AlpacaEval evaluation set (Dubois et al., 2023) using stratified sampling with a fixed random seed.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dubois et al., 2023 source

  • May combine two claims.

    Findingown data

    The average absolute difference between the two benchmark judges' scores for identical responses was 0.54 points, with a median of 0.38 points and a standard deviation of 0.58 points.

    “The average absolute difference was 0.54 points, the median was 0.38 points, and the standard deviation was 0.58 points.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Three domain experts independently clustering 1,000 anonymized responses achieved Fleiss' κ = 0.64, indicating substantial agreement.

    “Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fleiss, 1971 source

  • The quote only partly states this claim.

    Findingown data

    Automated equal-quality pair classification agreed with human cluster-based labels at a rate of 79.8% on deliberately selected hard cases near the ε-boundary.

    “The resulting 79.8% agreement rate is obtained on deliberately selected hard cases near the ε-boundary.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qwen3-235B-A22B-Thinking-2507 exhibits the strongest self-preference on Text Generation tasks (β = 0.336), followed by LongCat-Flash-Chat (β = 0.319).

    “Qwen3-235B-A22B-Thinking-2507 (Yang et al., 2025) exhibits the strongest self-preference (β = 0.336), followed by LongCat-Flash-Chat (β = 0.319).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Three models are good evaluators: DeepSeek-V3-0324 (π = 0.82, β = 0.024), Grok-4-Fast (π = 0.85, β = 0.035), and Kimi-Linear-48B-A3B-Instruct (π = 0.85, β = -0.043).

    “Three models are good evaluators: DeepSeek-V3-0324 (DeepSeek-AI, 2024) (π = 0.82, β = 0.024), Grok-4-Fast (π = 0.85, β = 0.035), and Kimi-Linear-48B-A3B-Instruct (Moonshot AI, 2025c) (π = 0.85, β = -0.043).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites DeepSeek-AI, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Moonshot AI, 2025c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Qwen3-235B-A22B-Thinking-2507 exhibits very strong self-preference on Information Provision (β = 0.466), followed by LongCat-Flash-Chat (β = 0.270).

    “Qwen3-235B-A22B-Thinking-2507 exhibits very strong self-preference (β = 0.466), followed by LongCat-Flash-Chat (β = 0.270).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DeepSeek-V3-0324 combines top-tier generation scores with near-zero bias (β = 0.024), showing high capability can coexist with impartiality.

    “DeepSeek-V3-0324 combines top-tier generation scores with near-zero bias (β = 0.024), proving that high capability can coexist with impartiality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LongCat-Flash-Chat and DeepSeek-V3.2 combine strong average quality with relatively high positive SPB.

    “LongCat-Flash-Chat and DeepSeek-V3.2combine strong average quality with relatively high positive SPB.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Information Provision tasks show eight significant models with average SPB β = 0.21.

    “Information Provision shows eight significant models ( β = 0.21).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although overall quality is not significantly correlated with SPB, many high-quality models still exhibit notable bias.

    “However, although overall quality is not significantly correlated with SPB, many high-quality models still exhibit notable bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Six models systematically favor themselves despite high discriminability: LongCat-Flash-Chat (π = 0.87, β = 0.307), DeepSeek-V3.2 (π = 0.82, β = 0.226), Gemma-3-27B (π = 0.81, β = 0.152), Qwen3-235B-A22B-Thinking-2507 (π = 0.85, β = 0.124), Grok-3-Mini (π = 0.85, β = 0.100), and Qwen3-235B-A22B-2507 (π = 0.86, β = 0.090).

    “Six models systematically favor themselves despite high discriminability”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CoT can sometimes yield slightly higher SPB than the baseline, whereas the structured method reduces SPB for all four models in the comparison.

    “As shown in Table 7, CoT can sometimes yield slightly higher SPB than the baseline, whereas Ours reduces SPB for all four models in this comparison.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Four models combine high capability with robust negative self-preference: Claude-Sonnet-4.5 (π = 0.84, β = -0.229), Llama-3.3-70B-Instruct (π = 0.84, β = -0.151), Hunyuan-A13B-Instruct (π = 0.83, β = -0.152), and Kimi-Dev-72B (π = 0.84, β = -0.117).

    “Four models combine high capability with robust negative self-preference: Claude-Sonnet-4.5 (π = 0.84, β = -0.229),Llama-3.3-70B-Instruct(AI@Meta, 2024) (π = 0.84, β = -0.151),Hunyuan-A13B-Instruct (Tencent Hunyuan Team, 2025) (π = 0.83, β = -0.152),and Kimi-Dev-72B (Moonshot AI, 2025a) (π = 0.84, β = -0.117).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites AI@Meta, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Tencent Hunyuan Team, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Moonshot AI, 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Explanation tasks show seven models with significant SPB and average β = 0.25.

    “Explanation tasks show seven models with significant SPB ( β = 0.25).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude-Sonnet-4.5 is high-quality and exhibits significant negative bias (β = -0.229).

    “Claude-Sonnet-4.5, while also high-quality, exhibits significant negative bias (β = -0.229)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Seven models are classified as Incompetent Randomizers: Llama-3.2-3B-Instruct (π = 0.51, β = -0.001), Llama-3.1-8B-Instruct (π = 0.77, β = -0.060), Mistral-Nemo (π = 0.75, β = -0.052), Gemma-3-12B (π = 0.78, β = 0.181), GLM-4.5-Air (π = 0.52, β = 0.095), DeepSeek-R1-0528 (π = 0.55, β = -0.097), and Kimi-K2-Thinking (π = 0.51, β = -0.102).

    “Incompetent Randomizers.This includes Llama-3.2-3B-Instruct(AI@Meta, 2024) (π = 0.51, β = -0.001),Llama-3.1-8B-Instruct(AI@Meta, 2024) (π = 0.77, β = -0.060),Mistral-Nemo (Mistral AI, 2024) (π = 0.75, β = -0.052),Gemma-3-12B (Gemma Team & Google DeepMind, 2025) (π = 0.78, β = 0.181), GLM-4.5-Air(Zhipu AI & GLM Team, 2025) (π = 0.52, β = 0.095), DeepSeek-R1-0528 (DeepSeek-AI, 2025a) (π = 0.55, β =”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites AI@Meta, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Mistral AI, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites DeepSeek-AI, 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The structured multi-dimensional mitigation strategy yields a statistically significant reduction in bias across all four subjects, with average β decreasing by 31.5%.

    “The strategy yields a statistically significant reduction in bias across all subjects, with the average β i decreasing by 31.5%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among the 20 models, 8 display positive self-preference (β > 0), 9 display negative self-preference, and 3 have near-zero SPB values.

    “In total, 8 models display positive selfpreference (β > 0), 9 models display negative self-preference, and 3 models-Grok-4-Fast (xAI, 2025b) (β = 0.035), DeepSeek-V3-0324 (DeepSeek-AI, 2024) (β = 0.024), and Llama-3.2-3B-Instruct(AI@Meta, 2024) (β = -0.001)havenear-zero SPB values.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites xAI, 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites DeepSeek-AI, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites AI@Meta, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Across 20 mainstream models, high capability does not necessarily imply fair evaluation; several highly discriminative models still show substantial SPB.

    “Across 20 mainstream models, we find that high capability does not necessarily imply fair evaluation: several highly discriminative models still show substantial SPB.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The paper identifies and formalizes a previously hidden class of Machiavellian Judges, models with high discriminability that systematically favor their own outputs.

    “We identify and formalize a previously hidden class of Machiavellian Judges, demonstrating that high model capability does not necessarily imply evaluative objectivity.”
    • Also stated in: § 3.5. Discriminability-bias classification · plain
      “Machiavellian Judges: π i ≥ 0.8 and β i > 0.08.These models possess high discriminability but systematically favor their own outputs, rendering them unreliable as judges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The structured multi-dimensional evaluation preserves high-contrast discriminability while reducing SPB.

    “rely on empirical validation (SPB decreases while highcontrast discriminability is preserved; see Section 4.4).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Among the evaluated models, LongCat-Flash-Chat has the highest positive self-preference bias (SPB).

    “Table C.19 reports SPB values for two representative models-one with the highest positive SPB (LongCat-Flash-Chat) and one with the strongest negative SPB (Claude-Sonnet-4.5)-across a range of ε values from 0 to 0.75.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude-Sonnet-4.5 exhibits robust negative self-preference bias (SPB) that is stable across all tested equal-quality thresholds ε from 0 to 0.75.

    “The results demonstrate that the main conclusions-LongCat-Flash-Chat exhibits robust positive SPB and Claude-Sonnet-4.5 exhibits robust negative SPB-are stable across all tested thresholds, supporting the robustness of the framework to the choice of ε.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Among the evaluated models, Claude-Sonnet-4.5 has the strongest negative self-preference bias (SPB).

    “Table C.19 reports SPB values for two representative models-one with the highest positive SPB (LongCat-Flash-Chat) and one with the strongest negative SPB (Claude-Sonnet-4.5)-across a range of ε values from 0 to 0.75.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    The two benchmark judges, GPT-5-Chat-Latest and Gemini-2.5-Pro, were selected because they provide stable high-quality judging behavior and strong agreement across large-scale scoring, and they are used exclusively for quality assessment rather than response generation.

    “These two models were selected because they provide stable high-quality judging behavior and strong agreement across largescale scoring, and they are used exclusively for quality assessment rather than response generation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The paper introduces a fully automated framework for SPB quantification and mitigation that operates without human gold standards and explicitly resolves the conflation between model capability and evaluative bias.

    “We design a framework for SPB quantification and mitigation, which operates without human gold standards and explicitly resolves the conflation between model capability and evaluative bias.”
    • Also stated in: § Abstract · plain
      “we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The two benchmark judges demonstrate statistically significant consistency and share the capability to reliably distinguish high-quality from low-quality responses.

    “The results yield ρ = 0.658, demonstrating statistically significant consistency between the two judges and confirming their shared capability to reliably distinguish high-quality from low-quality responses.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The 100-sample setting may underpower fine-grained conclusions for certain task types with limited representation.

    “the 100-sample setting, while sufficient for aggregate analysis, may underpower fine-grained conclusions for certain task types with limited representation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 100-task sample is sufficient for aggregate analysis of SPB across models.

  • May combine two claims.

    Interpretationnone given

    These Incompetent Randomizer models lack fundamental competence, rendering their SPB values practically meaningless.

    “They lack fundamental competence, rendering their SPB values practically meaningless.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Subsequent self-preference analysis is restricted exclusively to models satisfying π_i ≥ 0.8.

    “In this paper, we adopt π i ≥ 0.8 as the empirical threshold for discriminability, and subsequent self-preference analysis is restricted exclusively to models satisfying this criterion.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The framework provides a scalable solution suitable for deployment in automated evaluation systems across diverse expert system domains.

    “This framework provides a scalable solution suitable for deployment in automated evaluation systems across diverse expert system domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The absolute bias threshold |β_i|_thresh = 0.08 is a conservative cutoff that lies clearly above the noise scale (roughly 3-4 × σβ).

    “We therefore set |β i | thresh = 0.08 as a conservative cutoff that lies clearly above the noise scale (roughly 3-4 × σ β ).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The selected 20 evaluated models span both closed-source and open-source models across parameter scales from 3B to 671B and cover dense, Mixture-of-Experts, and reasoning-enhanced architectures.

    “we selected a diverse set spanning both closed-source and opensource models across a wide range of parameter scales from 3B to 671B.The selection covers distinct architectures including dense models, Mixture-of-Experts (MoE), and reasoning-enhanced models (e.g., Thinking series).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Study design and conductreasoning

    The double-query design, presenting each pair in both orders, avoids misinterpreting a model's preference for a specific position as a genuine preference for the content of a response.

    “The purpose of this mechanism is to avoid misinterpreting a model's preference for a specific position (i.e., position bias) as a genuine preference for the content of a response.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Family effects are not fully disentangled from pure self-preference, making it difficult to precisely separate whether observed biases stem from self-identification or shared stylistic and training characteristics across related models.

    “family effects (i.e., systematic preferences toward models from the same model family or lineage, such as models developed by the same organization) are not fully disentangled from pure self-preference, making it difficult to precisely separate whether observed biases stem from self-identification or shared stylistic and training characteristics across related models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    All task-specific analyses in Section 4.2 are restricted to the three sufficiently sampled categories: Text Generation, Information Provision, and Explanation.

    “Accordingly, all task-specific analyses in Section 4.2 are restricted to the three sufficiently sampled categories: Text Generation, Information Provision, and Explanation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    All model API calls used identical inference parameters (temperature=0, top_p=1) to eliminate confounding effects from parameter differences.

    “To eliminate confounding effects from parameter differences, all model API calls use identical inference parameters (temperature=0, top_p=1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The paper's framework constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards.

    “we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The design draws on cognitive science evidence that humans have limited working memory and tend to use heuristic shortcuts under high multi-attribute load, so decomposition can reduce extraneous load by forcing dimension-wise deliberation.

    “The design draws on cognitive science evidence that humans have limited working memory and tend to use heuristic shortcuts under high multi-attribute load so the decomposition can reduce extraneous load by forcing dimension-wise deliberation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sweller, 1988 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The experiments are primarily English-centric, which may limit generalizability to multilingual or domain-specific settings.

    “our experiments are primarily English-centric, which may limit the generalizability of the findings to multilingual or domain-specific settings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The neutral, single-letter output prompt eliminates confounding factors.

    “To eliminate confounding factors, we design a neutral, single-letter output prompt (see Appendix A,Table A.16).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The measurements are defined relative to benchmark judges rather than an absolute notion of objectivity, which may introduce residual dependence on the judging behavior of these reference models.

    “our measurements are defined relative to benchmark judges rather than an absolute notion of objectivity, which may introduce residual dependence on the judging behavior of these reference models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The three near-zero models do not reach significance, confirming that their observed SPB values are indistinguishable from sampling noise.

    “the three near-zero models do not, confirming that their observed SPB values are indistinguishable from sampling noise.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A model's SPB is deemed statistically significant only when at least two of the three tests (Binomial, Z-test, Bootstrap) agree.

    “we adopt a unified”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The design ensures the singular role of benchmark judges, avoids potential family bias by excluding the benchmark judge families from the evaluated set, and allows the remaining models to act as both generators and self-judges in the evaluation stage.

    “This design ensures the singular role of benchmark judges, avoids potential family bias by excluding the benchmark judge families from the evaluated set, and allows the remaining models to act as both generators and self-judges in the evaluation stage.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    DeepSeek-V3-0324 is classified as an Objective Judge.

    “| DeepSeek-V3-0324 | 0.82 0.024 < 10 -10 | Objective Judge |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Grok-4-Fast is classified as an Objective Judge.

    “| Grok-4-Fast | 0.85 0.035 < 10 -12 | Objective Judge |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    DeepSeek-R1-0528 is classified as an Incompetent Randomizer.

    “| DeepSeek-R1-0528 | 0.55 -0.097 | 0.184 | Incompetent Randomizer |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    LongCat-Flash-Chat is classified as a Machiavellian Judge.

    “| LongCat-Flash-Chat | 0.87 0.307 < 10 -14 | Machiavellian Judge |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Grok-3-Mini is classified as a Machiavellian Judge.

    “| Grok-3-Mini | 0.85 0.100 < 10 -12 | Machiavellian Judge |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Kimi-K2-Thinking is classified as an Incompetent Randomizer.

    “| Kimi-K2-Thinking | 0.51 -0.102 | 0.457 | Incompetent Randomizer |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    GLM-4.5-Air is classified as an Incompetent Randomizer.

    “| GLM-4.5-Air | 0.52 0.095 | 0.381 | Incompetent Randomizer |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Llama-3.2-3B-Instruct is classified as an Incompetent Randomizer.

    “| Llama-3.2-3B-Instruct | 0.51 -0.001 | 0.460 | Incompetent Randomizer |”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmark-defined higher-quality response is a valid indicator of response quality.
54 of 54

  • The authors state this as an assumption.

    Score differences of ≤0.25 between the two benchmark judges' composite scores for two responses reflect measurement noise rather than true quality disparity.

    “Given the scoring step of 0.25, differences satisfying |s i,q -s j,q | ≤ 0.25 are treated as measurement noise rather than true quality disparity.”

  • The paper relies on this without support.

    A benchmark score gap of at least 2.5 points guarantees that the higher-scored response is significantly superior in quality.

    “This condition guarantees that r j,q is significantly superior to r k,q in terms of quality.”

  • The paper relies on this without support.

    Human expert clustering based on perceived quality equivalence is a valid gold standard for equal-quality pair validation.

    “Three domain experts independently clustered these responses based on perceived quality equivalence, with final cluster assignments determined by majority vote (≥ 2/3).”

  • The paper relies on this without support.

    The two benchmark judges' scores provide a reliable quality benchmark for all evaluated models.

    “The entire framework rests on a reliable quality benchmark for model responses.”

  • The paper relies on this without support.

    The benchmark judges' scores provide a valid reference for response quality, so that equal-quality pairs and PIR/Null-PIR differences isolate self-preference bias.

    “our measurements are defined relative to benchmark judges rather than an absolute notion of objectivity, which may introduce residual dependence on the judging behavior of these reference models.”

  • The paper relies on this without support.

    The 100-task sample is sufficient for aggregate analysis of SPB across models.

    “the 100-sample setting, while sufficient for aggregate analysis, may underpower fine-grained conclusions for certain task types with limited representation.”

  • The paper relies on this without support.

    The task categories derived from root-verb distribution are meaningful and stable for task-specific analyses.

    “Following the root-verb distribution analysis, the subset covers seven categories: Text Generation (36%), Information Provision (34%), Explanation (16%), Question Answering (2%), Code Implementation (3%), Text Editing (3%), and Other tasks (6%).”

  • The paper relies on this without support.

    The benchmark-defined higher-quality response is a valid indicator of response quality.

    “whether the model identifies the benchmark-defined higher-quality response more often than chance.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Qwen Thinking task-level SPB values disagree with Table 4

    Major

    Table 4 lists this model's Text Generation, Information Provision, and Explanation SPB values as 0.86, 0.54, and 0.47. The accompanying discussion instead gives 0.336 for Text Generation, 0.466 for Information Provision, and 0.540 for Explanation.

  • LongCat Text Generation SPB disagrees with Table 4

    Major

    Table 4 gives LongCat-Flash-Chat a Text Generation SPB of 0.43, whereas the discussion reports 0.319 for that task.

  • The claimed pointwise baseline is a pairwise judgment

    Minor

    The mitigation section says the baseline scores responses independently on five dimensions. Its cited baseline prompt instead presents two responses together and requests a single A/B preference, so the described baseline is not the one shown in the prompt template.

  • Gemma-3-12B mitigation reduction has two values

    Minor

    The mitigation figure labels Gemma-3-12B's reduction as 26.1%, while Table 7 reports 26.5% for the same setting.

  • Structured prompt does not specify the stated majority-vote verdict

    Minor

    The methods say the final verdict is determined by majority vote over the five dimensions. The actual prompt instead instructs the judge to decide based on overall quality, without specifying majority vote.

  • Binomial test is described as both one-sided and two-sided

    Minor

    The methods specify a one-sided Binomial test, while the Table 3 caption says all three reported tests are two-sided.

6 of 6

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T.
Affiliations
University of Electronic Science and Technology of China (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2604.22891
arXiv
2604.22891
OpenAlex
W7156814473
Cited by
0
References
63
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Natural Science Foundation of China (T2293771, 42361144718, STI2030)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
LLM-as-a-judge, automated assessment, bias quantification, bias mitigation, multidimensional assessment, large language models, model alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review