CoChat
Sign Up
PreprintCited by 0Machine Learning and Data ClassificationTopic ModelingMachine Learning and Algorithms

Decomposing LLM-Judge Uncertainty to Target Expert Labels

Open access
Lail, Ryan·2026·arXiv
DOI 10.48550/arxiv.2609.06444arXiv 2609.06444
Open this paper in CoChat — free accountOpen PDF

Abstract

An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling. Code and data are available at https://github.com/composo-ai/judge-uncertainty-decomposition.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes decomposing LLM-judge uncertainty into aleatoric uncertainty (real expert-pool disagreement) and epistemic uncertainty (the judge’s ignorance), using a small Bayesian regression on already-collected labels with closed-form components. On ChaosNLI, epistemic ranking removes 15.0 units of error versus entropy’s 8.2 at a 10% escalation budget, though a fewest-labels-first rule ties it at 14.3.

Key findings

  • On a real LLM judge over dice with known truth, the estimated components isolate: aleatoric estimate correlates 0.45 with true disagreement and −0.12 with true error; epistemic estimate correlates 0.08 with true disagreement and 0.66 with true error, while the two truths are anti-correlated at −0.23 (Table 1).
  • Stated confidence is not a reliable escalation signal: by dice description tier, judge error is 0.53 with confidence 90.9 (none), 0.54 with 0.4 (weak), 0.41 with 59.1 (directional), and 0.007 with 14.8 (strong); item-level confidence correlates positively with error (Spearman +0.22), and escalating on low confidence does no better than random.
  • On ChaosNLI, the judge’s mean Manhattan error is 0.458 against a 0.158 noise ceiling, while stated confidence averages 90.3; at B = 10%, epistemic I-ranking removes 15.0 units of error versus entropy’s 8.2 (+6.8 [+1.7, +11.9]), 83% more per expert label, but fewest-labels-first removes 14.3 (+6.1 [−0.0, +12.4]) and is not beaten.
  • In calibrated simulations, entropy-ranking loses 34.8pp when only revealed label counts differ, the advantage is 0.0pp in the true null, and posterior spread captures ∼98% of the gain; in the misspecified simulation the advantage over entropy-ranking is ∼50pp except at γ = 1, where it is 3.6pp.

Methods

  • The model places a Dirichlet belief centered on the judge’s predicted distribution qx with trust λx fitted log-linearly on item features; labels update it to Dir(αx), with αx = λx qx + cx and α0,x = λx + nx. The decomposition is H(mx) = E[H(px) | cx] + I(y; px | cx), where the epistemic term is BALD’s acquisition score.
  • Two experiments use one real judge, Azure OpenAI’s gpt-5.6-terra, queried 18 August 2026. Experiment 1 uses 500 three-sided dice with known biases, four description tiers, 0–10 revealed rolls, three prompt variants, and 0–100 stated confidence. Experiment 2 uses 3,113 ChaosNLI items, each labeled by 100 annotators, with labels half-split per item, a 0.158 noise ceiling, and evaluation on the untouched half.
  • Escalation rules rank items by estimated uncertainty and escalate the top B% for a fixed budget; Experiment 2 reports B = 10% with 200 label re-draws. Baselines include entropy, verbalised confidence, fewest-labels-first, random, and oracle; component isolation is measured by Spearman correlations.

Limitations

  • The paper states that the full method is demonstrated only on synthetic dice; on ChaosNLI the fitted trust λ was nearly flat (sd(log λ) = 0.22 against the planned ≥ 0.5), so only the label-count side of the model was really tested on real data, and when trust is flat the ranking follows label counts, exactly what fewest-labels-first does. It also states each experiment used one judge model and one prompt design, simulated effect sizes depend on the generator, intervals resample labels over a fixed item set and say nothing about new items, and there is no defence against a deceptive judge that games the familiarity signals.
  • The text makes evident that the planned headline Δ-ranking was inconclusive against entropy (+4.5 [−2.2, +9.4]) and the counts-only claim was refuted (+6.9, grazing zero at −0.4); seven rules are compared, so no interval stands alone.
  • The paper states that separating the estimator from the count heuristic needs items genuinely differing in judge familiarity with dense labels, and no public dataset offers both—described as the field’s missing dataset rather than this paper’s missing experiment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2609.06444

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 6 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    At item level, stated confidence correlates positively with error (Spearman +0.22).

    “At item level it correlates positively with error (Spearman +0.22)”
    Quote in the paperSubstantial claim
  • Findingreasoning

    In simulations, when evidence is equal the gap between total uncertainty and epistemic ranking is exactly zero.

    “When evidence is equal the gap is exactly zero, as the Sec. 3 claim requires.”
    Quote in the paperSubstantial claim
  • Findingown data

    On ChaosNLI, simply escalating the least-labelled items does as well as the epistemic ranking.

    “though simply escalating the least-labelled items does as well there.”
    • Also stated in: § Contributions. · plain
      “though not more than a count-based rule.”
    Quote in the paperSubstantial claim
  • Findingown data

    The judge's stated confidence is no guide to its actual error.

    “stated confidence is no guide to its actual error”
    • Also stated in: § Contributions. · plain
      “the judge's stated confidence, taken at face value, is no guide to its actual error”
    • Also stated in: § 5.2 Experiment 1: a real LLM judge on dice with exactly known truth · plain
      “Stated confidence fails as an escalation signal: near-certainty with no information, near-zero when merely told the dice are weighted, calibrated only near complete information (Table 3).”
    Quote in the paperSubstantial claim
  • Findingown data

    On ChaosNLI, the epistemic ranking removes 83% more error than total uncertainty for the same expert labels.

    “On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 41
  • May combine two claims. 45
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    The epistemic term I_x is BALD's acquisition score.

    “The epistemic term I x is BALD's acquisition score [Houlsby et al., 2011]”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Houlsby et al., 2011 source
    • cites Bickford Smith et al., 2023 source
    • cites Nguyen et al., 2019 source

      Section 3 defines this paper’s epistemic uncertainty measure as u_e(x) and uses it to select instances, but does not identify an I_x term or BALD’s acquisition score. Searches for BALD, Houlsby, and information gain found no passage in our copy addressing that identification. The printed sentence cites Houlsby et al. (2011), not the supplied Nguyen et al. paper. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Withholding and purchasing optimise different scores, both distinct from entropy.

    “Withholding and purchasing optimise different scores [Bickford Smith et al., 2023], both distinct from entropy.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Bickford Smith et al., 2023 source

      Section 2.1 describes purchasing a label as maximising an acquisition function, and Section 4 distinguishes EPIG from BALD; the paper also treats predictive entropy as a separate baseline (Additional information). Our copy does not describe withholding as an optimisation decision or identify a score for it, so it does not support the claimed withholding–purchasing contrast. The passages provide no page numbers. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    On a real LLM judge against exactly known truth, the two estimated components isolate: each tracks its own ground truth and not the other's.

    “The components isolate on a real LLM judge against exactly known truth”
    • Also stated in: § 5.2 Experiment 1: a real LLM judge on dice with exactly known truth · plain
      “On a real LLM judge the two components are separable: each estimated component tracks its own ground truth and not the other's (Table 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In simulations, when items differ only in how many labels they already hold, ranking by total uncertainty forfeits a third of the error reduction the same labels could have bought, while epistemic ranking stays at oracle level.

    “When items differ only in how many labels they already hold, ranking by total uncertainty forfeits a third of the error reduction the same labels could have bought, while epistemic ranking stays at oracle level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Each prediction is worth about five pseudo-labels.

    “The fitted trust is near-uniform, sd(log λ) = 0.22 against the planned condition's ≥ 0.5, each prediction worth about five pseudo-labels.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    For the none tier, judge error is 0.53 and stated confidence is 90.9.

    “none | 0.53 | 90.9”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A judge-independent familiarity signal of middling quality recovers most of it (23.5pp) and costs nothing elsewhere.

    “A judge-independent familiarity signal of middling quality (orange) recovers most of it (23.5pp)and costs nothing elsewhere.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The fitted coefficient on stated confidence flips from -0.63 to +1.09 at γ = 1.5.

    “(fitted coefficient on stated confidence flips from -0.63 to +1.09)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Entropy (posterior) removes 6.9 (0.1) Manhattan units of error.

    “entropy (posterior) | 6.9 (0.1) | -1.2 [-6.0, +3.3]”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For the strong tier, judge error is 0.007 and stated confidence is 14.8.

    “strong | 0.007 | 14.8”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The gap between epistemic ranking and entropy widens at larger budgets and is absent at smaller ones.

    “The gap widens at larger budgets and is absent at smaller ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Fewest-labels-first captures part of the gap and plateaus.

    “fewest-labels-first captures part of the gap and plateaus”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The judge's mean Manhattan error is 0.458 against the 0.158 ceiling.

    “The judge's mean Manhattan error is 0.458 against the 0.158 ceiling, and stated confidence averages 90.3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Posterior spread v has an advantage over entropy of +3.9 [-1.7, +10.7].

    “ours: posterior spread v | 12.1 (0.2) | +3.9 [-1.7, +10.7]”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Random has an advantage over entropy of +0.1 [-5.5, +5.6].

    “random | 8.3 (0.2) | +0.1 [-5.5, +5.6]”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Item-level Spearman with error is 0.11 for prediction entropy.

    “Item-level Spearman with error is -0.13 for stated confidence and 0.11 for prediction entropy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Item-level Spearman with error is -0.13 for stated confidence.

    “Item-level Spearman with error is -0.13 for stated confidence and 0.11 for prediction entropy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The value clusters are count-aware 12.1-15.0 and count-blind 6.9-8.3.

    “The value clusters are count-aware 12.1-15.0and count-blind 6.9-8.3, and I minus fewest-labels-first is +0.7 [-6.3, +7.1].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the same label budget, epistemic ranking removes 15.0 units of error against entropy's 8.2, which is 83% more per expert label.

    “For the same label budget, epistemic ranking removes 15.0 units of error against entropy's 8.2 (Table 2): 83% more per expert label.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    In calibrated simulation, est-epistemic vs true disagreement is -0.14.

    “calibrated simulation 0.77 (.76,.78) | 0.13 | -0.14 | 0.48 (.46,.51) | -0.03”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    This judge predicts uniform exactly when ignorant, so ranking by total uncertainty nearly coincides with ranking by ignorance, and between-rule differences drown in single-draw noise.

    “this judge predicts uniform exactly when ignorant, so ranking by total uncertainty nearly coincides with ranking by ignorance, and between-rule differences drown in single-draw noise (App.C).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Verbalised confidence removes 8.1 (0.1) Manhattan units of error.

    “verbalised confidence | 8.1 (0.1) | -0.1 [-5.8, +5.1]”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Ranking by posterior spread captures ∼98% of the gain (v-ranking within 1.2-1.8pp of ∆-ranking everywhere).

    “(4) Ranking by posterior spread captures ∼98% of the gain (v-ranking within 1.2-1.8pp of ∆-ranking everywhere).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The collapse requires both uninformative confidence and silently varying quality.

    “The prediction that a regression using only revealed-label counts survives all tested corruptions was refuted at γ = 1 and confirmed elsewhere: the collapse requires both uninformative confidence and silently varying quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    I-ranking beats entropy by +12.8 [+4.3, +20.4] at B = 20%, with smaller budgets inconclusive.

    “I-ranking beats entropy by +12.8 [+4.3, +20.4] at B = 20%, with smaller budgets inconclusive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    I minus fewest-labels-first is +0.7 [-6.3, +7.1].

    “The value clusters are count-aware 12.1-15.0and count-blind 6.9-8.3, and I minus fewest-labels-first is +0.7 [-6.3, +7.1].”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Entropy-ranking (on the judge's output, or on the posterior with labels folded in) degrades from 35pp toward random's 85pp as the spread of item trust grows at B = 10%.

    “Entropy-ranking (on the judge's output, or on the posterior with labels folded in) degrades from 35pp toward random's 85pp”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Verbalised confidence has an advantage over entropy of -0.1 [-5.8, +5.1].

    “verbalised confidence | 8.1 (0.1) | -0.1 [-5.8, +5.1]”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The fitted trust is near-uniform, sd(log λ) = 0.22 against the planned condition's ≥ 0.5.

    “The fitted trust is near-uniform, sd(log λ) = 0.22 against the planned condition's ≥ 0.5, each prediction worth about five pseudo-labels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Stated confidence averages 90.3.

    “The judge's mean Manhattan error is 0.458 against the 0.158 ceiling, and stated confidence averages 90.3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Random removes 8.3 (0.2) Manhattan units of error.

    “random | 8.3 (0.2) | +0.1 [-5.5, +5.6]”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The fitted coefficient grows through γ = 0.75, reaches zero at γ = 1, and flips sign at inversion.

    “The prediction that the regression would monotonically down-weight stated confidence as corruption grows was refuted in its monotone form: the fitted coefficient grows through γ = 0.75, reaches zero at γ = 1, and flips sign at inversion.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    For the weak tier, judge error is 0.54 and stated confidence is 0.4.

    “weak | 0.54 | 0.4”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For the directional tier, judge error is 0.41 and stated confidence is 59.1.

    “directional | 0.41 | 59.1”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On ChaosNLI, neither the judge's stated confidence nor its prediction entropy tracks its error.

    “Neither confidence nor prediction entropy tracks error (App.D).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Findingown data

    Entropy (posterior) has an advantage over entropy of -1.2 [-6.0, +3.3].

    “entropy (posterior) | 6.9 (0.1) | -1.2 [-6.0, +3.3]”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The prediction that a regression using only revealed-label counts survives all tested corruptions was refuted at γ = 1 and confirmed elsewhere.

    “The prediction that a regression using only revealed-label counts survives all tested corruptions was refuted at γ = 1 and confirmed elsewhere: the collapse requires both uninformative confidence and silently varying quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pass/fail criteria: advantage ≥ 15pp at the criterion cell and oracle tracking Spearman ≥ 0.9, measured 80.9pp (seed-level CI 80.6-81.2) and 1.0.

    “Pass/fail criteria: advantage ≥ 15pp at the criterion cell and oracle tracking Spearman ≥ 0.9, measured 80.9pp (seed-level CI 80.6-81.2) and 1.0.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When shown its own rolls, the judge's self-updated prediction (error 0.258, n = 380) loses to conjugate updating of its own prior once the familiarity channel sets the trust level (0.262 with the channel, 0.307 without).

    “When shown its own rolls, the judge's self-updated prediction (error 0.258, n = 380) loses to conjugate updating of its own prior (adding the observed rolls to its stated prediction as counts) once the familiarity channel sets the trust level (0.262 with the channel, 0.307 without)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Total predictive entropy tracks judge error at 0.57, against 0.66 for the epistemic estimate.

    “Total predictive entropy tracks judge error at 0.57, against 0.66 for the epistemic estimate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Entropy sees only the shape of m_x, not the evidence behind it: with equal total evidence the two rankings nearly agree, and where evidence varies they come apart, up to full inversion.

    “Entropy sees only the shape of m x , not the evidence behind it: with equal total evidence the two rankings nearly agree, and where evidence varies they come apart, up to full inversion (App.B).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Both uncertainty components follow as simple formulas, with no sampling or further judge calls.

    “Both components follow as simple formulas, with no sampling or further judge calls.”
    • Also stated in: § Contributions. · plain
      “Both components follow as simple formulas with no further judge calls.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The judge's prediction is never fitted; the judge does the perceptual work, and the model decides how far to believe it.

    “The judge's prediction is never fitted: the judge does the perceptual work, our model decides how far to believe it.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The method has no defence against a deceptive judge that games the familiarity signals behind its trust.

    “We have no defence against a deceptive judge that games the familiarity signals behind its trust.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    None of the prior work asks which labels to buy.

    “None asks which labels to buy.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    In the calibrated simulation (2,000 items), the world is drawn from the Sec. 3 model with the judge's prior correct by construction, so the value of every escalation decision is exactly computable.

    “In the calibrated simulation (2,000 items), the world is drawn from the Sec. 3 model with the judge's prior correct by construction, so the value of every escalation decision is exactly computable.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Correctly predicting a genuinely split pool gives D_x = 0: disagreement is not error.

    “Correctly predicting a genuinely split pool gives D x = 0: disagreement is not error.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Only the ∆ row was the planned headline comparison.

    “Only the ∆ row was the planned headline comparison.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Dubois et al. fit generalised linear models to grader scores: theirs separates grader severity offsets from residual disagreement, while ours separates irreducible pool disagreement from our own ignorance of where the pool sits.

    “Dubois et al. [2026] fit generalised linear models to grader scores: theirs separates grader severity offsets from residual disagreement, while ours separates irreducible pool disagreement from our own ignorance of where the pool sits.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dubois et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Seven rules are compared, so no interval stands alone.

    “Seven rules are compared, so no interval stands alone.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    In the misspecified simulation, the world is drawn truth-first, the judge's prediction quality varies with familiarity, and its stated confidence is corrupted by a knob γ: honest at γ = 0, information-free at γ = 1, inverted at γ = 1.5.

    “In the misspecified simulation, the world is drawn truth-first, the judge's prediction quality varies with familiarity, and its stated confidence is corrupted by a knob γ: honest at γ = 0, information-free at γ = 1, inverted at γ = 1.5.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The full method is demonstrated only on synthetic dice, where the truth is exactly known.

    “We demonstrate the full method only on synthetic dice, where the truth is exactly known.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Labels are half-split per item, fitting and acquisition on one half, evaluation the other: evaluation truth is never touched by training.

    “Labels are half-split per item, fitting and acquisition on one half, evaluation the other: evaluation truth is never touched by training”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    In the misspecified simulation, the estimator's model is wrong by construction and rules are scored against an oracle that knows the realised world, so only between-rule comparisons are meaningful.

    “There the estimator's model is wrong by construction and rules are scored against an oracle that knows the realised world, so only between-rule comparisons are meaningful.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each experiment used one judge model and one prompt design.

    “Each experiment used one judge model and one prompt design.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    When trust is flat, the ranking follows label counts, exactly what the fewest-labels-first baseline does, so the baseline ties the method.

    “But when trust is flat our ranking follows label counts, exactly what the fewest-labels-first baseline does, so the baseline ties us.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Simulated effect sizes depend on the generator.

    “Simulated effect sizes depend on the generator (Sec.5.1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Entropy is blind to evidence and, through revealed counts, anti-correlated with label value.

    “(2) Entropy-ranking approaches random as trust spread grows (81pp regret vs random's 85pp at the widest setting), because entropy is blind to evidence and, through revealed counts, anti-correlated with label value: the claim's inversion, realised.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    No new data are needed.

    “No new data are needed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    These magnitudes are generator-dependent: they do not predict real-world effect sizes.

    “These magnitudes are generator-dependent: they do not predict real-world effect sizes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The aleatoric component is the entropy of the pool's population label distribution, a property of the pool unreduced by labels.

    “Its entropy H(p x ) (or Gini G(p x ) = 1 -∥p x ∥ 2 ) is the aleatoric component: a property of the pool, unreduced by labels.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Annotators in a pool hold label distributions over classes, and labels are draws from those distributions.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The intervals resample labels over a fixed item set and say nothing about new items.

    “Our intervals resample labels over a fixed item set and say nothing about new items.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    The remaining advantages are exploratory.

    “The remaining advantages are exploratory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The planned condition tested only the trust side and so predicted no separation, a mis-specification: the characterisation is about α0, not about λ alone.

    “The planned condition tested only the trust side and so predicted no separation, a mis-specification: the characterisation is about α 0 , not about λ alone.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The calibrated simulation draws worlds from the Sec. 3 model itself, so the value of every escalation choice is exactly computable.

    “The calibrated one draws worlds from the Sec. 3 model itself, so the value of every escalation choice is exactly computable.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Total uncertainty H(m_x) is the escalation baseline.

    “total uncertainty H(m x ) is the escalation baseline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    On ChaosNLI, label counts range 0-10 by design, leaving total evidence α0 heterogeneous where λ is flat.

    “But label counts range 0-10 by design, leaving total evidence α 0 heterogeneous where λ is flat (the Sec. 3 condition).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The level-2 belief is a Dirichlet centred on an LLM judge's predicted distribution q_x, weighted by a trust λ_x in pseudo-label units.

    “The level-2 belief is a Dirichlet centred on an LLM judge's predicted distribution qx , weighted by a trust λ x in pseudo-label units.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The level-2 belief is a Dirichlet centred on the LLM judge's predicted distribution q_x, weighted by a trust λ_x in pseudo-label units.

  • May combine two claims.

    Interpretationreasoning

    Separating the estimator from the count heuristic needs items genuinely differing in judge familiarity, with dense labels to score.

    “Separating our estimator from the count heuristic needs items genuinely differing in judge familiarity, with dense labels to score.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Where total evidence varies, the two rankings come apart, folded-in labels driving entropy anti-correlated with value, up to full inversion.

    “Where it varies they come apart, folded-in labels driving entropy anti-correlated with value, up to full inversion.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The entropy display of Sec. 3 is the categorical counterpart of the three-way variance decomposition, with the first two terms merged by the Level-0 identifiability constraint.

    “The entropy display of Sec. 3 is its categorical counterpart, with the first two terms merged by the Level-0 identifiability constraint above.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Entropy sees only the shape of m_x, but label value also sees the evidence behind it, decaying as (α_0,x + 1)^-2.

    “Entropy sees only the shape of m x , but label value also sees the evidence behind it, decaying as (α 0,x + 1) -2 .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Equal evidence is the only α_0 condition giving agreement for every shape, and it fails in any partially-labelled deployment: label counts alone make α_0 heterogeneous.

    “Equal evidence is the only α 0 condition giving agreement for every shape, and it fails in any partially-labelled deployment: label counts alone make α 0 heterogeneous.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The first two terms of the three-way variance decomposition are aleatoric (irreducible for a single fresh label) and the third alone shrinks with labels.

    “whose first two terms are aleatoric (irreducible for a single fresh label) and whose third alone shrinks with labels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Neither μ_a(x) nor τ^2(x) is directly observable; a label carries the annotator's own noise with it.

    “Neither µ a (x) nor τ 2 (x) is directly observable: a label carries the annotator's own noise with it.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Annotator noise is centred, so averaging labels recovers m(x).
    • The authors state this as an assumption: Within-annotator noise and between-annotator disagreement are both aleatoric and can be summed.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    A handful of coefficients against a few hundred labelled items is well-posed: only the trust scale is learned.

    “A handful of coefficients against a few hundred labelled items is well-posed: only the trust scale is learned.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Both within-annotator noise and between-annotator disagreement are aleatoric, and every quantity in the paper uses only their sum.

    “both are aleatoric, and every quantity in the paper uses only their sum.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Within-annotator noise and between-annotator disagreement are both aleatoric and can be summed.

  • May combine two claims.

    Interpretationreasoning

    A label's variance is τ^2(x) + σ^2 with σ^2 = sum_a w_a σ_a^2, so one label per (item, annotator) pair cannot separate genuine disagreement from within-annotator noise.

    “a label's variance is τ 2 (x) + σ2 with σ2 = a w a σ 2 a , so one label per (item, annotator) pair cannot separate genuine disagreement from within-annotator noise.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Annotator noise is centred, so averaging labels recovers m(x).
    • The authors state this as an assumption: Within-annotator noise and between-annotator disagreement are both aleatoric and can be summed.
78 of 78

  • The authors state this as an assumption.

    Annotators in a pool hold label distributions over classes, and labels are draws from those distributions.

    “Annotator a in a pool with weights w a holds a label distribution π a (x) over the K classes, and a label is a draw from it.”

  • The authors state this as an assumption.

    The pool's population label distribution p_x is never observed, so we hold a belief about it.

    “p x is never observed, so we hold a belief about it.”

  • The authors state this as an assumption.

    The level-2 belief is a Dirichlet centred on the LLM judge's predicted distribution q_x, weighted by a trust λ_x in pseudo-label units.

    “The level-2 belief is a Dirichlet centred on an LLM judge's predicted distribution qx , weighted by a trust λ x in pseudo-label units.”

  • The authors state this as an assumption.

    The judge may be ignorant of where the pool's distribution sits.

    “The judge may also be ignorant of where the pool's distribution sits: epistemic uncertainty.”

  • The paper relies on this without support.

    No public dataset offers both items genuinely differing in judge familiarity and dense labels.

    “No public dataset offers both: the field's missing dataset rather than this paper's missing experiment.”

  • The authors state this as an assumption.

    Annotators hold a latent score µ_a(x) ∈ [0,1], the score they would settle on given unlimited care.

    “Annotator a holds a latent score µ a (x) ∈ [0, 1], the score they would settle on given unlimited care, and an observed label adds within-annotator noise, which we take to be Gaussian, y = µ a (x) + ε, ε ∼ N (0, σ 2 a ), where σ 2 a is the variance of a fresh rating of the same item by the same annotator.”

  • The authors state this as an assumption.

    Observed labels add within-annotator Gaussian noise.

    “Annotator a holds a latent score µ a (x) ∈ [0, 1], the score they would settle on given unlimited care, and an observed label adds within-annotator noise, which we take to be Gaussian, y = µ a (x) + ε, ε ∼ N (0, σ 2 a ), where σ 2 a is the variance of a fresh rating of the same item by the same annotator.”

  • The authors state this as an assumption.

    τ^2(x) is genuine inter-annotator disagreement, a property of the pool itself, unreduced by collecting further labels.

    “where τ 2 (x) is genuine inter-annotator disagreement, a property of the pool itself, unreduced by collecting further labels.”

  • The paper relies on this without support.

    Annotator noise is centred, so averaging labels recovers m(x).

    “Averaging labels still recovers m(x), since that noise is centred”

  • The authors state this as an assumption.

    Within-annotator noise and between-annotator disagreement are both aleatoric and can be summed.

    “both are aleatoric, and every quantity in the paper uses only their sum.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Lower reported error is described as a loss

    Major

    The judge's self-updated prediction has error 0.258, lower than the 0.262 reported for conjugate updating with the familiarity channel. On the stated error metric, the self-updated prediction does not lose that comparison.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. Dubois, Coppock, et al. Quantifying biases in LLM-as-a-judge evaluations. In ICML, 2026.
  2. Plank. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In EMNLP, 2022.

Related

Authors
Lail, R.
Corresponding author
Lail, Ryan
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2609.06444
arXiv
2609.06444
OpenAlex
W7212044799
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
LLM-as-a-judge, uncertainty decomposition, epistemic uncertainty, aleatoric uncertainty, Bayesian regression, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review