Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Probably not a substantial claim: may not pass S3 (load-bearing).

A common case is rubric-based evaluation, where a holistic judgment is decomposed into independently graded criteria, each typically binary (MET/UNMET).

“A common case is rubricbased evaluation, where a holistic judgment is decomposed into independently graded criteria, each typically binary (MET/UNMET) (Hashemi et al., 2024;Saad-Falcon et al., 2023).”

• cites Hashemi et al., 2024 sourcePartly supported

  The LLM-RUBRIC Method describes assessing an overall judgment through separate questions about evaluation criteria. But Evaluation Rubric Construction says the criteria usually have scores of 1–4, not binary MET/UNMET responses; it also does not establish that criteria are graded independently. Page numbers are unavailable in our copy. (LLM, conf=0.89)

  Experimental — accuracy not yet measured
• cites Saad-Falcon et al., 2023 sourcePartly supported

  Section 3.2 describes three separately evaluated RAG criteria, each graded by a binary classifier; Section 3.3 describes using those labels to score systems. The paper supports this as an example of separate binary evaluation, but does not establish that rubric-based evaluation commonly decomposes holistic judgments or uses MET/UNMET labels. Page numbers are unavailable in our copy. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Analyses of human-judge disagreement (Plank, 2022) suggest some criterion-level disagreement is signal rather than noise.

“analyses of human-judge disagreement (Plank, 2022) suggest some criterion-level disagreement is signal rather than noise.”

• cites Plank, 2022 sourcePartly supported

  The Introduction (page not available in our copy) says some human-label disagreement reflects plausible variation rather than error and argues that this variation contains information worth retaining. The paper does not establish the more specific claim about disagreement between judges on individual criteria. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent NLP work develops the two-axis view for question answering, classification, and calibrated LLM uncertainty.

“recent NLP work develops the same view for question answering (Kamath et al., 2020), classification (Xin et al., 2021), and calibrated LLM uncertainty (Kadavath et al., 2022).”

• cites Kamath et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xin et al., 2021 sourceNot checked — no full text of the source
• cites Kadavath et al., 2022 sourceSupported

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

What current papers do not articulate is what these choices cost.

“What current papers do not articulate is what these choices cost.”

• AssumptionRests on an assumption: The 24 surveyed papers are representative of current practice in LLM-as-judge evaluation.

May combine two claims.

With three values {0, 1/2, 1}, the rank map is affine in the input value iff n0 = n1 - iff MET and UNMET prevalences coincide.

“Direct calculation shows this map is affine in the input value iff n 0 = n 1 -iff MET and UNMET prevalences coincide.”

May combine two claims.

From the identity, α ≥ κ_F with equality iff κ_F = 1, and α → κ_F as N → ∞.

“so α ≥ κ F with equality iff κ F = 1, and α → κ F as N → ∞.”

Probably not a substantial claim: may not pass S4 (based).

These relationships are rarely connected to how LLM judges are scored.

“but they are rarely connected to how LLM judges are scored”

• AssumptionRests on an assumption: The 24 surveyed papers are representative of current practice in LLM-as-judge evaluation.

May combine two claims.

The average pairwise Pearson, Spearman, Kendall's τ_b, φ, and MCC over the R^2 judge pairs are all equal, and coincide with κ_F when all judges share the same marginal MET rate.

“The average pairwise Pearson, Spearman, Kendall's τ b , ϕ, and MCC over the R 2 judge pairs are all equal, and coincide with κ F when all judges share the same marginal MET rate.”

• cites Conger (1980) sourceNot checked — no full text of the source

May combine two claims.

Cohen's κ is close to the common value of Section 4 but not equal to it.

“Cohen's κ is close to the common value of Section 4 but not equal to it.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

With N = 100 and (TP, FN, FP, TN) = (40, 10, 20, 30), π = 0.50 and π = 0.60, direct computation gives Acc = 0.700, F1 = 0.727, p_e = 0.500, κ = 0.400, and φ ≈ 0.408.

“With N = 100 and (T P, F N, F P, T N ) = (40, 10, 20, 30), π = 0.50 and π = 0.60.Direct computation gives Acc = 0.700, F 1 = 0.727, p e = 0.500, κ = 0.400, and ϕ = 1000/ √ 6 × 10 6 ≈ 0.408.”

May combine two claims.

At the balanced matched point (p11, p10, p01, p00) = (0.4,0.1, 0.1, 0.4) which adds π = 0.5 on top of matched marginals, κ = φ = 0.6, and the multinomial delta method using the closed-form κ variance of Fleiss et al. (1969) gives Var(κ) = Var(φ) = 16/(25N) = 0.640/N - not the 0.41/N that the bivariate normal approximation (1 - φ2)2/N would yield for continuous Pearson, which does not apply to

“At the balanced matched point (p 11 , p 10 , p 01 , p 00 ) = (0.4,0.1, 0.1, 0.4)which adds π = 0.5 on top of matched marginals κ = ϕ = 0.6, and the multinomial delta method using the closed-form κ variance of Fleiss et al. (1969) gives Var(κ) = Var( φ) = 16/(25N ) = 0.640/N -not the 0.41/N that the bivariatenormal approximation (1 -ϕ 2 ) 2 /N would yield for continuous Pearson, which does not”

• cites Fleiss et al. (1969)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

May combine two claims.

In a two-criterion rubric where criterion A fires on 80% of items and is easy (κ = 0.9) and criterion B fires on 5% of items and is hard (κ = 0.2), micro-κ is dominated by A's decisions, landing near 0.85; macro-κ averages to 0.55.

“Consider a twocriterion rubric in which criterion A fires on 80% of items and is easy (κ = 0.9) and criterion B fires on 5% of items and is hard (κ = 0.2).Micro-κ is dominated by A's decisions, landing near 0.85; macroκ averages to 0.55;”

May combine two claims.

At the matched point (0.3, 0.1, 0.1, 0.5) where π = π = 0.4, the cell probabilities have unequal row and column sums while the marginals π and π remain equal; direct computation gives ∇κ - ∇φ = (5/6) 1, confirming the all-ones structure away from the π = 0.5 balanced case.

“As a numerical check, at the matched point (0.3, 0.1, 0.1, 0.5) -where π = π = 0.4 -the cell probabilities have unequal row and column sums while the marginals π and π remain equal; direct computation gives ∇κ -∇ϕ = (5/6) 1, confirming the all-ones structure away from the π = 0.5 balanced case.”

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

May combine two claims.

|κ| ≤ |φ|, with equality iff A = B, which is equivalent to π = π.

“and |κ| ≤ |ϕ|, (12) with equality iff A = B, which is equivalent to π = π.”

May combine two claims.

For typical values (κ_F = 0.6, N = 100, R = 3), α - κ_F ≈ 1.3 × 10^-3, below the precision at which agreement coefficients are reported.

“For typical values (κ F = 0.6, N = 100, R = 3), α -κ F ≈ 1.3 × 10 -3 , below the precision at which agreement coefficients are reported.”

May combine two claims.

The handling of ties, invalid outputs, refusals, and insufficient-information (CANNOT_ASSESS) responses is heterogeneous-sometimes excluded, sometimes folded into a valid label-and almost never reported as a deliberate choice.

“The handling of ties, invalid outputs, refusals, and insufficient-information (CANNOT_ASSESS) responsesfoot_0 is heterogeneoussometimes excluded, sometimes folded into a valid label -and almost never reported as a deliberate choice.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The one-vs-rest MET-class F1 here is 2 • 30/(2 • 30 + 15 + 15) = 60/90 ≈ 0.667, which numerically matches the recode-as-UNMET row by coincidence on this table - the two are different statistics whose values happen to agree here.

“The one-vs-rest MET-class F 1 here is 2 • 30/(2 • 30 + 15 + 15) = 60/90 ≈ 0.667, which numerically matches the recode-as-UNMET row by coincidence on this table -the two are different statistics whose values happen to agree here.”

May combine two claims.

In a low-prevalence regime (TP, FN, FP, TN) = (5, 5, 0, 90) (π = 0.10), Acc = 0.95 but κ ≈ 0.643 and φ ≈ 0.688.

“In a low-prevalence regime (T P, F N, F P, T N ) = (5, 5, 0, 90) (π = 0.10), Acc = 0.95 but κ ≈ 0.643 and ϕ ≈ 0.688;”

May combine two claims.

In the balanced case π = π = 0.5, p_e = 0.5 and κ = 2 Acc - 1.

“In the balanced case π = π = 0.5, p e = 0.5 and κ = 2 Acc -1.”

May combine two claims.

F1 does not depend on TN and is asymmetric under label flipping (MET ↔ UNMET).

“F 1 does not depend on T N and is asymmetric under label flipping (MET ↔ UNMET).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Table A1 gives the full audit summarized in Section 1: the 24 surveyed LLM-as-judge and LLM-evaluator systems, coded by judged output scale, reported metrics, and primary relevance category.

“Table A1 gives the full audit summarized in Section 1: the 24 surveyed LLM-as-judge and LLMevaluator systems, coded by judged output scale, reported metrics, and primary relevance category.”

May combine two claims.

Item-level κ depends on the rubric's weights and threshold, and may track either criterion or neither.

“item-level κ depends on the rubric's weights and threshold, and may track either criterion or neither.”

May combine two claims.

Every symmetric statistic in this paper-accuracy, F1, κ, φ/MCC, and the rank correlations-is invariant under FP ↔ FN, so none distinguishes a judge that misses positives from one that over-predicts them.

“As noted in Section 3, every symmetric statistic in this paper -accuracy, F 1 , κ, ϕ/MCC, and the rank correlations -is invariant under F P ↔ F N , so none distinguishes a judge that misses positives from one that over-predicts them.”

Probably not a substantial claim: may not pass S4 (based).

Abstention is treated as a preprocessing detail rather than a modeling choice.

“abstention is treated as a preprocessing detail rather than a modeling choice”

• AssumptionRests on an assumption: The 24 surveyed papers are representative of current practice in LLM-as-judge evaluation.

May combine two claims.

Off matched marginals the two variances diverge by a second-order term: at (0.4, 0.1, 0.2, 0.3) (π = 0.5, π = 0.6, κ = 0.400, φ = 1/√6 ≈ 0.408), the delta method gives Var(κ) = 504/(625N) ≈ 0.806/N and Var(φ) = 13/(16N) = 0.813/N.

“Off matched marginals the two variances diverge by a second-order term: at (0.4, 0.1, 0.2, 0.3) (π = 0.5, π = 0.6, κ = 0.400, ϕ = 1/ √ 6 ≈ 0.408), the delta method gives Var(κ) = 504/(625N ) ≈ 0.806/N and Var( φ) = 13/(16N ) = 0.813/N .”

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

May combine two claims.

Micro-, macro-, and item-level aggregation estimate different quantities, and a scalar agreement figure reported without naming its aggregation level is ambiguous.

“These numbers estimate different quantities, and a scalar agreement figure reported without naming its aggregation level is ambiguous.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Standard cluster-bootstrap implementations are available in the survey-statistics literature.

“standard cluster-bootstrap implementations are available in the survey-statistics literature (e.g., Field and Welsh, 2007).”

• cites Field and Welsh, 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

A multiclass setting still admits per-class F1, macro-F1, and micro-F1 aggregations, none of which is the same statistic as the binary F1 in the first two rows.

“a multiclass setting still admits per-class F 1 , macro-F 1 , and micro-F 1 aggregations, none of which is the same statistic as the binary F 1 in the first two rows.”

May combine two claims.

The identities below are exact for binary vectors; they do not make Pearson, Spearman, and Kendall redundant on graded, ordinal, or continuous scores.

“The identities below are exact for binary vectors; they do not make Pearson, Spearman, and Kendall redundant on graded, ordinal, or continuous scores.”

May combine two claims.

Under a plain misclassification loss, the CA and UNMET responses become indistinguishable when recoding CA as UNMET, but the calibration information the CA verdict was introduced to carry is destroyed at the measurement stage.

“Under a plain misclassification loss the CA and UNMET responses become indistinguishable, but the calibration information the CA verdict was introduced to carry is destroyed at the measurement stage.”

May combine two claims.

Average pairwise Spearman and τ_b on three-valued data no longer coincide; their divergence is governed by the per-item distribution of CA votes across judges.

“Average pairwise Spearman and τ b on threevalued data no longer coincide; their divergence is governed by the per-item distribution of CA votes across judges.”

May combine two claims.

The slice-orthogonal direction (0, +1, -1, 0) is the p10 ↔ p01 swap, under which both metrics are symmetric and stationary, so the two four-dimensional gradients can differ only along the simplex normal 1.

“The slice-orthogonal direction (0, +1, -1, 0) is the p 10 ↔ p 01 swap, under which both metrics are symmetric and stationary, so the two fourdimensional gradients can differ only along the simplex normal 1.”

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

May combine two claims.

Abstention handling is not preprocessing but a choice of estimand - excluding, recoding, and retaining CANNOT_ASSESS each estimate a different quantity, and the last breaks the binary equivalences.

“Abstention handling is not preprocessing but a choice of estimand -excluding, recoding, and retaining CANNOT_ASSESS each estimate a different quantity, and the last breaks the binary equivalences.”

May combine two claims.

The three abstention-handling modes estimate different quantities, corresponding to different questions about the judge.

“They estimate different quantities, corresponding to different questions about the judge.”

May combine two claims.

The phi coefficient has no canonical 3×3 analog; the multi-class MCC extension of Gorodkin (2004) does not reduce to either Pearson on a chosen encoding or to Kendall's τb.

“The phi coefficient has no canonical 3 × 3 analog; the multi-class MCC extension of Gorodkin (2004) does not reduce to either Pearson on a chosen encoding or to Kendall's τ b .”

• cites Gorodkin (2004) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The declarations (judgment scale, abstention and tie handling mode, coverage, confusion matrix, aggregation level) cost no more than the counts a validation already computes, and together they let readers reconstruct the equivalent statistics and compare across studies.

“None costs more than the counts a validation already computes, and together they let readers reconstruct the equivalent statistics and compare across studies.”

May combine two claims.

The mechanism is a gradient-coincidence argument on the simplex, not a cell-by-cell partial-derivative identity, and the variance gap off matched marginals is second-order in the mismatch.

“The mechanism is a gradient-coincidence argument on the simplex, not a cell-by-cell partialderivative identity, and the variance gap off matched marginals is second-order in the mismatch; Appendix C gives the argument and a numerical check.”

May combine two claims.

Admitting CANNOT_ASSESS breaks the equivalences of Observation 2 in three ways, each by the mechanism of Section 6.2.

“Admitting CANNOT_ASSESS breaks the equivalences of Observation 2 in three ways, each by the mechanism of Section 6.2.”

Probably not a substantial claim: may not pass S2 (checkable).

The relationships that settle such questions are classical.

“The relationships that settle such questions are classical (Cohen, 1960;Matthews, 1975;Warrens, 2008)”

• cites Cohen, 1960 sourceNot checked — no full text of the source
• cites Matthews, 1975 sourceNot checked — no full text of the source
• cites Warrens, 2008Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For binary vectors, the Pearson correlation reduces to the phi coefficient, which is identical to the binary Matthews Correlation Coefficient.

“which is identical to the binary Matthews Correlation Coefficient.”

• cites Matthews, 1975 sourceNot checked — no full text of the source

May combine two claims.

Cohen's κ adds information not as a fifth coefficient but through its chance-correction: the κ-φ gap measures how far the judge's positive rate has drifted from the human's, closing in value and, to leading order, in sampling variance exactly when the two rates coincide.

“Cohen's κ adds information not as a fifth coefficient but through its chance-correction: the κ-ϕ gap measures how far the judge's positive rate has drifted from the human's, closing in value and, to leading order, in sampling variance exactly when the two rates coincide.”

May combine two claims.

On the matched-marginals slice {p10 = p01}, both metrics collapse to the same function of cell probabilities, κ = φ = 1 - p10/(p1• p0•), so ∇κ and ∇φ agree along the two tangent directions of that slice.

“On the matchedmarginals slice {p 10 = p 01 }, both metrics collapse to the same function of cell probabilities, κ = ϕ = 1 -p 10 /(p 1• p 0• ), so ∇κ and ∇ϕ agree along the two tangent directions of that slice.”

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

Probably not a substantial claim: may not pass S2 (checkable).

The first reporting decision is not which coefficient to compute but what was being measured.

“their common lesson is that the first reporting decision is not which coefficient to compute but what was being measured.”

May combine two claims.

Average pairwise Pearson on three-class data requires a numeric encoding of {MET, UNMET, CA}; the result depends on the encoding, and no encoding is canonical.

“Average pairwise Pearson requires a numeric encoding of {MET, UNMET, CA}; the result depends on the encoding (e.g., {1, 0, 1/2} versus deleting CA pairs), and no encoding is canonical.”

Probably not a substantial claim: may not pass S2 (checkable).

The gap is not missing mathematics but a missing account of what the reported numbers mean on the scales LLM judges actually produce.

“The gap is not missing mathematics but a missing account of what the reported numbers mean on the scales LLM judges actually produce.”

• AssumptionRests on an assumption: The 24 surveyed papers are representative of current practice in LLM-as-judge evaluation.

Classified as probably not a substantial claim.

The Section 5.1 aggregation options (micro, macro, item-level) answer different questions when criteria differ in prevalence or difficulty.

“and the Section 5.1 aggregation options (micro, macro, item-level) answer different questions when criteria differ in prevalence or difficulty.”

• AssumptionRests on an assumption: Rubric evaluation is nested—criteria within items, items within prompts, possibly repeated judgments within models or ensembles.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Section 7 multi-judge treatment covers the standard pairwise-aggregable agreement statistics on the unweighted scale; it does not characterize the finite-sample behavior of weighted-κ ensembles under arbitrary weight matrices, nor the behavior of judge ensembles whose members are not exchangeable.

“The Section 7 multi-judge treatment covers the standard pairwise-aggregable agreement statistics on the unweighted scale; we do not characterize the finite-sample behavior of weighted-κ ensembles under arbitrary weight matrices, nor the behavior of judge ensembles whose members are not exchangeable (for example, one anchored on a frontier model and the rest on small open-weight models).”

May combine two claims.

On binary data the same scalar can stand for several distinct statistics, and its value shifts with the judgment scale, the handling of ties and abstentions, and the aggregation level.

“on binary data the same scalar can stand for several distinct statistics, and its value shifts with the judgment scale, the handling of ties and abstentions, and the aggregation level.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The |κ| < |φ| gap is small because the marginal mismatch is small.

“The |κ| < |ϕ| gap is small because the marginal mismatch is small.”

May combine two claims.

Rubric evaluation is not a flat sequence of independent decisions; the same hierarchy that underlies the cluster-bootstrap caveat forces a second choice, about how to aggregate the per-decision metric into a single number.

“Rubric evaluation is not a flat sequence of independent decisions; the same hierarchy that underlies the cluster-bootstrap caveat forces a second choice, about how to aggregate the per-decision metric into a single number.”

• AssumptionRests on an assumption: In rubric evaluation, criteria are clustered within items, within prompts or rubrics, and within judges.

May combine two claims.

Whether the counts admit a sampling-distribution interpretation depends on the inferential target: when the evaluation set is itself the object of measurement and the judge is run at temperature zero the counts are deterministic, while generalizing to a wider item population or accounting for judge stochasticity at non-zero temperature places them in the asymptotic regime of Section 5 (or, for有限样本

“Whether these counts additionally admit a sampling-distribution interpretation depends on the inferential target: when the evaluation set is itself the object of measurement and the judge is run at temperature zero the counts are deterministic, while generalizing to a wider item population or accounting for judge stochasticity at non-zero temperature places them in the asymptotic regime of Section”

• AssumptionThe authors state this as an assumption: The observed confusion-matrix counts are fixed.

May combine two claims.

The three modes of handling CA/tie/invalid estimate different quantities; a metric reported without its handling mode is ambiguous among them.

“As Section 6 shows, the three modes estimate different quantities; a metric reported without its handling mode is ambiguous among them.”

Classified as probably not a substantial claim. May combine two claims.

The exclude estimand is the selective-accuracy measure of the binary judge, restricted to the (1 - Pr[m=1])(1 - Pr[ m=1]) fraction of the population on which both sides chose to commit; it characterizes performance on the covered subset, not on the full deployment distribution.

“The exclude estimand is the selective-accuracy measure of the binary judge, restricted to the (1 -Pr[m = 1])(1 -Pr[ m = 1]) fraction of the population on which both sides chose to commit; it characterizes performance on the covered subset, not on the full deployment distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The analysis treats the observed confusion-matrix counts as fixed; the identities and inequalities hold exactly on any dataset on which they are computed.

“The analysis treats the observed confusion-matrix counts as fixed; the identities and inequalities hold exactly on any dataset on which they are computed.”

Classified as probably not a substantial claim. May combine two claims.

This work analyzes the relationships among agreement metrics and examines how they are reported in literature; it introduces no new data, models, or human-subjects experiments.

“This work analyzes the relationships among agreement metrics and examines how they are reported in literature; it introduces no new data, models, or human-subjects experiments.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The equivalence results concern the binary row; the nominal-with-abstention row is taken up in Section 6.

“The equivalence results concern the binary row; the nominal-with-abstention row is taken up in Section 6.”

May combine two claims.

Observation 1 also fails on binary data with degenerate marginals; the k = 3 break is qualitatively different in that it cannot be repaired by any choice of encoding.

“Observation 1 also fails on binary data with degenerate marginals; the k = 3 break is qualitatively different in that it cannot be repaired by any choice of encoding.”

Probably not a substantial claim: may not pass S2 (checkable).

The reporting practices recommended here are intended to make such comparisons more transparent.

“the reporting practices recommended here are intended to make such comparisons more transparent.”