Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

The difference between the two framings exceeds the spread among the three LLM judges only on FED-Turn, where it is 10.5 points against a spread of 2.9.

“The difference between the two framings exceeds the spread among the three LLM judges only on FED-Turn, where it is 10.5 points against a spread of 2.9”

On ELLIPSE the best judge, Gemini, reaches 29.8% exact accuracy where the reference reaches 51.4%.

“on ELLIPSE the best of them, Gemini, reaches 29.8% exact accuracy where the reference reaches 51.4%”

When one judge is wrong, another gives the same wrong verdict more often than independent errors would.

“when one judge is wrong, another gives the same wrong verdict more often than independent errors would”

Jev Score's exact accuracy is lower, without separation, on HelpSteer2 and USR-PC.

“Its exact accuracy is lower, without separation, on HelpSteer2 and USR-PC.”

Jev Choice finished the nine panels in 29 seconds of wall time.

“Jev Choice finished the nine panels in 29 seconds of wall time.”

On graded criteria all four judges agree more with one another than with the labels.

“On graded criteria all four judges agree more with one another than with the labels”

• Also stated in: § 1 Introduction · plain

  “On every graded panel the four judges' agreement with one another exceeds their agreement with the human labels”

• Also stated in: § 5.1 Judge-judge agreement exceeds judge-label agreement · plain

  “The judge-judge row of Table 2 exceeds the judgelabel row on all seven graded panels”

Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev.

“Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev”

• Also stated in: § 1 Introduction · plain

  “Over the nine panels Gemini's bill was 325 times Jev's”

Over the nine panels Jev Choice cost $0.063.

“Over the nine panels Jev Choice cost $0.063”

In exact accuracy, Jev Choice exceeds the rater reference on LFQA and USR-TC.

“in exact accuracy, Jev Choice exceeds it on LFQA and USR-TC”

Gemini is the most accurate, or tied for it, everywhere except LFQA.

“Gemini is the most accurate, or tied for it, everywhere except LFQA”

A post-hoc shift of each judge's levels on a criterion by a constant removes most of the shortfall against the raters.

“A post-hoc shift of each judge's levels on a criterion by a constant removes most of it”

Jev Noul and Jev Choice give nearly the same verdicts on binary criteria, with Cohen's κ = 0.945 between them over the 8,392 pairs of the full RiceChem set.

“Jev Noul and Jev Choice give nearly the same verdicts on binary criteria, with Cohen's κ = 0.945 between them over the 8,392 pairs of the full RiceChem set”

Every judge falls short of the rater reference on ELLIPSE, FED-Dialogue, and USR-PC.

“Every judge falls short of the rater reference on ELLIPSE, FED-Dialogue, and USR-PC”

Of the 21 comparisons on the graded panels, 16 do not separate Jev Choice from an LLM judge.

“Of the 21 comparisons on the graded panels, 16 do not separate Jev Choice from an LLM judge.”

Jev Choice ranks second on HealthBench, behind Gemini.

“and second on HealthBench, behind Gemini”

Jev Score is separated from Jev Choice on four graded panels (ELLIPSE, FED-Turn, FED-Dialogue, and USR-TC), each time in its own favour.

“Jev Score is separated from Jev Choice on four graded panels (ELLIPSE, FED-Turn, FED-Dialogue, and USR-TC), each time in its own favour”

Replayed post hoc with thresholds chosen on half of the units and scored on the rest, a cascade that falls back on the best LLM judge nearly matches that judge at under half its cost on most panels.

“Replayed post hoc with thresholds chosen on half of the units and scored on the rest, a cascade that falls back on the best LLM judge nearly matches that judge at under half its cost on most panels”

The LLM judges repeat nearly all of Jev's most confident errors.

“The LLM judges repeat nearly all of Jev's most confident errors”

No cascade's average gain over the best single judge of its panel, Jev's own framings included, exceeds 1.5 points.

“no cascade's average gain over the best single judge of its panel, Jev's own framings included, exceeds 1.5 points”

On single panels the cost ratio ranges from 18 (Luna on HealthBench) to 770 (Gemini on FED-Dialogue).

“On single panels the ratio ranges from 18 (Luna on HealthBench) to 770 (Gemini on FED-Dialogue).”

Most of the other comparisons between Jev and an LLM judge are inconclusive.

“most of the other comparisons are inconclusive”

The largest separation is Gemini's lead on ELLIPSE, 16.4 points (95% interval 14.1 to 18.7).

“The largest separation is Gemini's lead on EL-LIPSE, 16.4 points (95% interval 14.1 to 18.7).”

On FED-Turn Jev Score has the highest exact accuracy of any judge.

“Jev Score has the highest exact accuracy of any judge”

Jev Choice is never the most accurate matched judge on a graded panel.

“Jev Choice is never the most accurate matched judge on a graded panel.”

On six of the seven graded panels every judge's mean level lies below the labels' mean, by about a level on ELLIPSE's learner essays.

“on six of the seven every judge's mean level lies below the labels' mean, by about a level on ELLIPSE's learner essays”

The judges' shortfall against the raters is concentrated in a minority of criteria.

“Their shortfall against the raters is concentrated in a minority of criteria”

The judges' shortfall against the raters comes more from where a judge places units on the scale than from how it orders them.

“comes more from where a judge places units on the scale than from how it orders them”

Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons.

“Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons”

Jev's errors are repeated most often where it is most confident.

“Jev's errors are repeated most often where it is most confident”

Against LFQA's provenance baseline no judge's exact accuracy differs significantly in a pair-level sign test.

“Against LFQA's provenance baseline no judge's exact accuracy differs significantly in a pair-level sign test”

DeepSeek is never separated from Jev Choice.

“DeepSeek is never separated from Jev Choice”

Jev Score has the higher within-one accuracy on all seven graded panels.

“and has the higher within-one accuracy on all seven”

Summed over the nine panels, the LLM judges took 30 to 220 times as long as Jev.

“took 30 to 220 times as long”

• Also stated in: § 1 Introduction · plain

  “its run took 33 times as long”

The other 10 graded comparisons have 90% intervals that reach between 5.3 and 8.3 points from zero.

“The other 10 have 90% intervals that reach between 5.3 and 8.3 points from zero”

Jev Choice's probability for the NA option rises with the number of raters who answered N/A (Spearman ρ = 0.62 over 25 dialogues).

“Jev Choice's probability for the NA option rises with the number of raters who did (Spearman ρ = 0.62 over 25 dialogues)”

A cascade replayed on the recorded verdicts gains at most 2.0 points over the best single judge even with oracle thresholds.

“and at most 2.0 even with oracle thresholds”

For Gemini the cost ratio rises from 121 on HealthBench, with 2.0 criteria per unit, to its maximum on FED-Dialogue, with 10.

“For Gemini it rises from 121 on HealthBench, with 2.0 criteria per unit, to its maximum on FED-Dialogue, with 10”

Jev's confidence ranks its own errors on most panels.

“Jev's confidence ranks its own errors on most panels”

• Also stated in: § 1 Introduction · plain

  “on most panels its less confident verdicts err more often”

The median of the three LLM judges is never more accurate than the most accurate matched judge on any of the nine panels.

“The median is never more accurate than that judge on any of the nine panels.”

Jev Choice never predicts the middle level on FED-Turn correct and fluent, whose texts state yes-or-no properties.

“Jev Choice never predicts the middle level on FED-Turn correct and fluent, whose texts state yes-or-no properties (Appendix E).”

The LLM judges repeat Jev's wrong answers beyond what independent errors would produce, most strongly where Jev is most confident.

“yet the LLM judges repeat its wrong answers beyond what independent errors would produce, most strongly where Jev is most confident (Section 6.2).”

ELLIPSE is the only panel with a rater reference on which such a selection stays below it.

“ELLIPSE is the only panel with a rater reference on which such a selection stays below it”

LFQA's raters grant the top factuality level far more often than Jev Choice does.

“LFQA's raters grant the top factuality level far more often than Jev Choice does (Table 3).”

In the confident-error sample, the three LLM judges repeat Jev's wrong answer 96.0% of the time (242 of 252), where the independence baseline gives 50.3%.

“In their verdicts on these pairs, the three LLM judges repeat Jev's wrong answer 96.0% of the time (242 of 252), where the independence baseline gives 50.3%.”

Across the 31 graded criteria with a rater reference, the four matched judges' mean offset tracks their mean gap (Spearman ρ = 0.92).

“Across the 31 graded criteria with a rater reference (all but HelpSteer2's four), the four matched judges' mean offset tracks their mean gap (Spearman ρ = 0.92”

A fourth juror, Jev Choice, never improves significantly on the most accurate matched judge.

“A fourth juror, Jev Choice, never improves significantly on that judge either.”

On five of the seven graded panels even the least-agreeing pair of judges agrees more closely than the best judge agrees with the labels.

“and on five of them even the least-agreeing pair of judges agrees more closely than the best judge agrees with the labels”

Even with oracle thresholds, a cascade on ELLIPSE or FED-Dialogue must defer nearly every pair to match the best LLM judge.

“Even with oracle thresholds, a cascade on ELLIPSE or FED-Dialogue must defer nearly every pair to match the best LLM judge (Appendix K).”

USR-PC, where Jev keeps 10 of its 16 such pairs, has the largest gain among the graded panels.

“USR-PC, where Jev keeps 10 of its 16 such pairs, has the largest gain among the graded panels.”

Oracle thresholds reach the judge's accuracy on the same six panels at 10% to 59% of its cost.

“Oracle thresholds reach the judge's accuracy on the same six panels at 10% to 59% of its cost.”

On those six panels the AUROC of Jev Choice's confidence for its own correctness is 0.57 to 0.70, embedded binary pairs included with their Noul confidence.

“On those six panels the area under the receiver operating characteristic curve (AUROC) of its confidence for its own correctness is 0.57 to 0.70, embedded binary pairs included with their Noul confidence.”

Jev Score, the best single judge on FED-Turn, beats every FED-Turn cascade by at least 5.2 points.

“Jev Score, the best single judge on FED-Turn, beats every FED-Turn cascade by at least 5.2 points.”

On single traits Jev Choice's rank agreement with the raters is close to one rater's with the other.

“On single traits Jev Choice's rank agreement with the raters is close to one rater's with the other (Appendix G).”

In the two within-rubric contrasts of Section 5.4, ELLIPSE cohesion and HelpSteer2 verbosity, the judges come closest where a level description points to something a reader can see in the unit.

“In the two within-rubric contrasts of Section 5.4, ELLIPSE cohesion and HelpSteer2 verbosity, the judges come closest where a level description points to something a reader can see in the unit.”

The LLM judges abstain largely on the same pairs as Jev Choice.

“the LLM judges abstain largely on the same pairs”

The five criterion texts on the USR panels are byte-identical on Topical-Chat and PersonaChat.

“Their five criterion texts are byte-identical on Topical-Chat and PersonaChat.”

The judges rarely use the top of the scale.

“The judges rarely use the top of the scale (Figure 3a).”

The three-judge median returns Jev's wrong answer on 82 of the 84 pairs of the confident-error sample, and the label on one.

“The three-judge median also returns Jev's wrong answer on 82 of the 84 pairs of the confident-error sample, and the label on one (Appendix J).”

The mean of the option values that a judge chose for an essay's six traits correlates with the essay's holistic score at Spearman rho = 0.62 to 0.72 across the matched judges.

“The mean of the option values that a judge chose for an essay's six traits correlates with the essay's holistic score at Spearman ρ = 0.62 to 0.72 across the matched judges (Table C3d).”

Confidence also fails on single criteria inside panels where it otherwise ranks errors.

“Confidence also fails on single criteria inside panels where it otherwise ranks errors.”

Jev and Luna, the two judges run more than once, are not deterministic.

“Jev and Luna, the two judges run more than once, are not deterministic.”

With cut-offs chosen only to describe the pattern, 11 of the 12 criteria more than 15 points below the reference have a mean offset below -0.5 levels.

“With cut-offs chosen only to describe the pattern, 11 of the 12 criteria more than 15 points below the reference have a mean offset below -0.5 levels”

The trait cohesion, whose level descriptions name observable devices, is every judge's best.

“The trait cohesion, whose level descriptions name observable devices, is every judge's best.”

The two embedded binary criteria of USR-TC split in direction.

“The two embedded binary criteria of USR-TC split in direction.”

A post-hoc selection of whichever judge is right on each ELLIPSE pair would reach only the remaining 36.0%, below the rater reference.

“a post-hoc selection of whichever judge is right on each pair would reach only the remaining 36.0%,below the rater reference”

The shift leaves the judges' predictions spread over fewer levels than the labels.

“It leaves their predictions spread over fewer levels than the labels (Appendix G).”

Wherever one of these graded cascades beats its fallback, Jev and the fallback are within 2 points of each other on the kept pairs.

“Wherever one of these graded cascades beats its fallback, Jev and the fallback are within 2 points of each other on the kept pairs.”

Jev Choice's errors gather among its less confident verdicts on every graded panel but ELLIPSE.

“Jev Choice's errors gather among its less confident verdicts on every graded panel but ELLIPSE (Figure 4a).”

ELLIPSE has the largest offsets.

“ELLIPSE has the largest offsets.”

With cross-fitted thresholds, no cascade beats the best single judge by more than 1.5 points on average over the halvings, a margin reached on HealthBench.

“With cross-fitted thresholds, no cascade beats the best single judge by more than 1.5 points on average over the halvings, a margin reached on HealthBench (Table 4).”

Every matched judge also agrees best with the HelpSteer2 labels on verbosity, which describes an amount of detail visible in the response.

“Every matched judge also agrees best with the HelpSteer2 labels on verbosity, which describes an amount of detail visible in the response.”

For the oracle-threshold cascades of Table 4, kept pairs on which Jev alone is right are few on the graded panels, because on the pairs Jev keeps, Jev and the fallback are mostly right together and wrong together.

“For the oracle-threshold cascades of Table 4, kept pairs on which Jev alone is right are few on the graded panels, because on the pairs Jev keeps, Jev and the fallback are mostly right together and wrong together.”

In Jev's top confidence band, 80.5% to 92.8% of the LLM judges' verdicts on Jev's wrong pairs repeat its answer, where the baseline predicts 33.6% to 63.7%.

“There 80.5% to 92.8% of the LLM judges' verdicts on Jev's wrong pairs repeat its answer, where the baseline predicts 33.6% to 63.7%.”

Top levels worded as absolutes were read differently from the raters in both directions.

“Top levels worded as absolutes”

The LLM judges' MET-rate offsets have no common sign on the binary panels.

“where the LLM judges' METrate offsets have no common sign (Table 2).”

The cohesion trait still carries an offset of -0.52 to -0.62 levels.

“The cohesion trait still carries an offset of -0.52 to -0.62 levels”

Each of the 15 criteria whose mean offset lies within 0.3 levels of zero is at most 6 points below the reference.

“Each of the 15 criteria whose mean offset lies within 0.3 levels of zero is at most 6 points below the reference”

Of the 36 LLM verdicts on HealthBench's confident errors, 32 give Jev's answer, against 10.0 expected under the baseline.

“Of the 36 LLM verdicts on them, 32 give Jev's answer, against 10.0 expected under the baseline (Appendix J).”

Most confident errors sit on pairs whose raters disagree.

“Most confident errors sit on pairs whose raters disagree.”

Jev's graded offsets are negative outside USR-TC.

“Its graded offsets, by contrast, are negative outside USR-TC.”

Almost all of Jev Choice's ELLIPSE errors lie below the label.

“almost all of Jev Choice's ELLIPSE errors lie below it (Section 5.3).”

On most graded panels, the less confident Jev is, the more often it is wrong.

“On most graded panels, the less confident Jev is, the more often it is wrong (Section 6.1)”

Confidence still ranks errors on HelpSteer2 correctness, whose labels track overall helpfulness (AUROC 0.76).

“Confidence still ranks errors on HelpSteer2 correctness, whose labels track overall helpfulness (AUROC 0.76; Table I1e).”

Jev Choice is wrong more often on unanimous LFQA factuality pairs than on split ones.

“Jev Choice is wrong more often on unanimous LFQA factuality pairs than on split ones”

No median jury improves significantly on the most accurate matched judge.

“and no median jury improves significantly on the most accurate matched judge (Section 6.4).”

Jev over-grants MET on both binary panels.

“Jev over-grants MET on both binary panels (Appendix D)”

All four judges are wrong on 64.0% of ELLIPSE pairs.

“All four are wrong on 64.0% of ELLIPSE pairs”

The shifted judges are at most 5 points below the shifted rater reference.

“the shifted judges are at most 5 points below the shifted rater reference.”

The excess of repeated errors peaks in Jev's top confidence band on every panel.

“The excess peaks in Jev's top confidence band on every panel (Figure 4b).”

Every judge does worst on grammar, whose levels differ only by the quantifiers throughout, many, some, minimal, and few or no.

“Every judge does worst on grammar, whose levels differ only by the quantifiers”

Every judge's offset is negative on the six graded panels other than USR-TC.

“Every judge's offset is negative on the six graded panels other than USR-TC (Tables 2 and C3)”

Jev Choice, without the harness prompt, is nonetheless offset further on ELLIPSE than Gemini and DeepSeek.

“Jev Choice, without them, is nonetheless offset further on ELLIPSE than Gemini and DeepSeek (Table 2).”

Gemini's FED-Dialogue lead has a pair-level adjusted value of 0.19.

“has a pair-level adjusted value of 0.19”

A 466-word context-seeking criterion of 23 pairs gains the most, rising from 39.1% to 82.6%.

“A 466-word context-seeking criterion of 23 pairs gains the most, rising from 39.1% to 82.6%”

Jev Choice did not choose CANNOT_ASSESS on any of the 8,392 RiceChem pairs or the 406 HealthBench pairs.

“Jev Choice did not choose CANNOT_ASSESS on any of the 8,392 RiceChem pairs or the 406 HealthBench pairs.”

The intervals separate Luna's lead on USR-PC instead, whose pair-level p is 0.10.

“The intervals separate Luna's lead on USR-PC instead, whose pair-level p is 0.10.”

The largest loss is on a global-health criterion that is met when the response contains no inaccuracies that could lead to harm.

“The largest is on a global-health criterion that is met when the response contains no inaccuracies that could lead to harm.”

Among the uncorrected pair-level tests, Jev Score is significantly less accurate than an LLM judge in 2 of its 21 comparisons on ordinal pairs, both against Gemini, on ELLIPSE and HelpSteer2.

“significantly less accurate in 2, both against Gemini, on ELLIPSE and HelpSteer2”

The HelpSteer2 and LFQA labels partly reflect something other than the named criterion.

“The HelpSteer2 and LFQA labels partly reflect something other than the named criterion.”

Two other context-seeking criteria of similar length each lose 10.0 points.

“yet two other context-seeking criteria of similar length each lose 10.0 points”

Applied to the bootstrap p-values, Holm correction keeps four separations.

“Applied to the bootstrap p-values, it keeps four separations (Section 4.1)”

HelpSteer2 correctness tracks overall helpfulness at Pearson r = 0.94.

“HelpSteer2 correctness tracks overall helpfulness at Pearson r = 0.94, and provenance alone predicts LFQA labels about as well as the judges do on exact accuracy.”

The chosen Jev Noul threshold, 0.58, gives 81.5% on the 80% training portion of RiceChem.

“The chosen threshold, 0.58, gives 81.5% on the training portion.”

Among Gemini's ratios, RiceChem is the exception to the order described in Section 4.2.

“Among Gemini's ratios, RiceChem is the exception to the order described in Section 4.2.”

Gemini's FED-Dialogue lead survives Holm correction of the bootstrap p-values.

“Gemini's FED-Dialogue lead, which survives Holm correction of the bootstrap pvalues”

Jev Choice is ahead on HelpSteer2 against Luna in the Holm-corrected pair-level tests (p Holm = 0.011).

“Jev Choice ahead on HelpSteer2 against Luna (p Holm = 0.011)”

HelpSteer2's correctness label alone tracks helpfulness.

“although HelpSteer2's correctness label alone tracks helpfulness (Table 3)”

Jev Noul's error rate falls with the margin |P(MET)-0.5|, from 46.1% on the full RiceChem set and 49.2% on HealthBench in the band nearest 0.5 to below 10% at margins of 0.4 or more.

“Jev Noul's error rate falls with the margin |P (MET) -0.5|, from 46.1% on the full RiceChem set and 49.2% on HealthBench in the band nearest 0.5 to below 10% at margins of 0.4 or more”

Criterion length does not predict the direction of the effect, since the criteria that lose accuracy run from 32 to 481 words.

“Criterion length does not predict the direction of the effect, since the criteria that lose accuracy run from 32 to 481 words.”

The share of pairs with identical probabilities in all five passes ranged from 4.2% of RiceChem pairs under bare Noul to 35.8% under Jev Choice.

“The share of pairs with identical probabilities in all five passes ranged from 4.2% of RiceChem pairs under bare Noul to 35.8% under Jev Choice.”

The three largest losses, of 20.0 to 30.8 points, fall on criteria of 13 or fewer pairs.

“The three largest losses, of 20.0 to 30.8 points, fall on criteria of 13 or fewer pairs.”

Bare Noul predicts MET on 78.4% of the panel's pairs against a labelled rate of 59.5%.

“Bare Noul predicts MET on 78.4% of the panel's pairs against a labelled rate of 59.5%.”

Among the uncorrected pair-level tests, Jev Score is significantly more accurate than an LLM judge in 7 of its 21 comparisons on ordinal pairs.

“Jev Score is significantly more accurate than an LLM judge in 7 of its 21 comparisons on ordinal pairs”

Nearly half of the ELLIPSE labels depend on the rounding rule.

“Nearly half of the ELLIPSE labels depend on the rounding rule (Section 3.4).”

Gemini is ahead on ELLIPSE in the Holm-corrected pair-level tests.

“Gemini ahead on ELLIPSE”

Provenance alone predicts LFQA labels about as well as the judges do on exact accuracy.

“HelpSteer2 correctness tracks overall helpfulness at Pearson r = 0.94, and provenance alone predicts LFQA labels about as well as the judges do on exact accuracy.”

Over the seven graded panels Jev Choice and Jev Score cost about the same, $0.043 and $0.042.

“Over the seven graded panels Jev Choice and Jev Score cost about the same, $0.043 and $0.042.”

DeepSeek returned an empty response on one RiceChem pair and two HealthBench pairs, the three binary-panel failures of Section 3.5, which the harness did not flag.

“It did the same on one RiceChem pair and two HealthBench pairs, the three binary-panel failures of Section 3.5, which the harness did not flag (Table F1).”

For every framing, P(MET) separates MET from UNMET pairs with an AU-ROC of 0.80 to 0.86 on the two panels.

“For every framing, P(MET) separates MET from UNMET pairs with an AU-ROC of 0.80 to 0.86 on the two panels”

No LLM judge chose CANNOT_ASSESS on either panel.

“No LLM judge chose CANNOT_ASSESS on either panel (Table F1).”

Applied to the pair-level tests, Holm correction keeps three separations.

“applied to the pair-level tests it keeps three”

Under every framing, each change occurred on a pair whose mean probability of MET lay within 0.1 of 0.5.

“Under every framing, each change occurred on a pair whose mean probability of MET lay within 0.1 of 0.5 (Figure D1b shows Jev Noul).”

Jev Choice's probabilities over the three levels, with the NA probability removed and the rest renormalized, give 55.4% exact accuracy when read by Score's rule, the rounded expected level.

“Jev Choice's probabilities over the three levels, with the NA probability removed and the rest renormalized, give 55.4% exact accuracy when read by Score's rule, the rounded expected level.”

The Jaccard index between Jev Choice's FED-Dialogue abstentions and Gemini's is 0.73.

“Jaccard index -0.560.73 0.72”

• cites Table F1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The two graded framings' Jev Noul verdicts on embedded binary pairs differ on 4 of the 364 pairs.

“These verdicts differ on 4 of the 364 pairs and change a panel's binary accuracy only on USR-PC (Table C3f).”

• cites Table C3fNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

On the 215 disagreements, Jev Score is exact on 21.9%.

“On the 215 disagreements Jev Choice is exact on 8.4% and Jev Score on 21.9%.”

Averaged over pairs, Jev Choice puts more probability than Jev Score on 'No.' (0.316 against 0.205).

“Averaged over pairs, Jev Choice puts more probability than Jev Score on”

The unit-level intervals of Table C1 separate Jev Choice and Jev Score on USR-TC.

“the unit-level intervals of Table C1 also separate the two on USR-TC.”

On the remaining graded panels each framing lies inside the LLM judges' range.

“On the remaining graded panels each framing lies inside the range.”

Jev Choice abstained on all 7 dialogues with two or more N/A answers.

“Jev Choice abstained on all 7 dialogues with two or more N/A answers, on 7 of the 8 with one, and on 3 of the 10 with none.”

Jev Choice abstained on 3 of the 10 dialogues with no N/A answers.

“Jev Choice abstained on all 7 dialogues with two or more N/A answers, on 7 of the 8 with one, and on 3 of the 10 with none.”

These three criteria (error_recovery, HelpSteer2 correctness, LFQA factuality) hold 27 of Jev Choice's 28 abstentions.

“These three criteria hold 27 of its 28 abstentions.”

On the unit aggregate's rank correlation with the holistic score, Jev Choice has 0.603.

“where Jev Choice has 0.603”

Scored against the same leave-one-out targets as a rater, Jev Score reaches 66.3% exact accuracy.

“Scored against the same leave-one-out targets as a rater, Jev Score reaches 66.3% exact accuracy”

Jev Score's probabilities give 64.6% exact accuracy when read by argmax.

“Jev Score's probabilities give 64.6% when read by argmax”

The six ELLIPSE traits, whose raters need no shift, range from 1.1 points below the reference (vocabulary) to 5.6 points above.

“After the best shift, the 12 criteria far below the reference lie between 1.1 points below it and 13.8 points above, and the six ELLIPSE traits, whose raters need no shift (Section 5.3), range from 1.1 points below it (vocabulary) to 5.6 points above.”

Jev Choice abstained on 7 of the 8 dialogues with one N/A answer.

“Jev Choice abstained on all 7 dialogues with two or more N/A answers, on 7 of the 8 with one, and on 3 of the 10 with none.”

Jev Choice's mean probability on the NA option is 0.57 on error_recovery.

“Its mean probability on the NA option is 0.57 on error_recovery, 0.11 on HelpSteer2 correctness, 0.023 on LFQA factuality, and below 0.02 on each of the other 32 graded criteria.”

Jev Score rises above the LLM judges' range on FED-Turn and USR-TC.

“Jev Score rises above it on FED-Turn and USR-TC.”

On topheavy three-level scales, a rater's best shift is often not zero.

“On topheavy three-level scales a rater's best shift is often not zero.”

After the best shift, the 12 criteria far below the reference lie between 1.1 points below it and 13.8 points above.

“After the best shift, the 12 criteria far below the reference lie between 1.1 points below it and 13.8 points above, and the six ELLIPSE traits, whose raters need no shift (Section 5.3), range from 1.1 points below it (vocabulary) to 5.6 points above.”

The 50 pairs with three or more physicians, from 20 completions, are free of this selection.

“The 50 pairs with three or more physicians, from 20 completions, are free of this selection.”

Most N/A answers report that no system error occurred.

“Because most N/A answers report that no system error occurred, the numeric ratings likely come mainly from raters who saw an error to recover from.”

On the 215 disagreements, Jev Choice is exact on 8.4%.

“On the 215 disagreements Jev Choice is exact on 8.4% and Jev Score on 21.9%.”

Jev Choice used only the NA option, the second of the two abstain options.

“The binary and graded panels offered different abstain options, and Jev Choice used only the second.”

Most of the LLM judges' graded abstentions fall on pairs where Jev Choice also abstained.

“Most of each LLM judge's graded abstentions are on FED-Dialogue, nearly all of those on error_recovery, and most of them fall on pairs where Jev Choice also abstained (Table F1 gives the Jaccard indices).”

Most of each LLM judge's graded abstentions are on FED-Dialogue.

“Most of each LLM judge's graded abstentions are on FED-Dialogue, nearly all of those on error_recovery, and most of them fall on pairs where Jev Choice also abstained (Table F1 gives the Jaccard indices).”

On HelpSteer2, four of Gemini's five abstentions are pairs on which Jev Choice also abstained.

“On HelpSteer2 four of Gemini's five abstentions are pairs on which Jev Choice also abstained.”

Averaged over pairs, Jev Choice puts less probability than Jev Score on 'Yes.' (0.513 against 0.585).

“and”

A single crowd worker reaches 63.8% exact accuracy against the same leave-one-out targets.

“and a single crowd worker 63.8% (Table E1).”

Every judge places ELLIPSE essays below their labels.

“Because every judge places ELLIPSE essays below their labels (Section 5.3), the framing that answers one level higher gains exact accuracy on its disagreements.”

The twelfth criterion far below the reference, FED-Turn engaging, has a mean offset of -0.34 levels.

“The twelfth criterion far below the reference, FED-Turn engaging, has a mean offset of -0.34 levels.”

The 11 criteria of Section 5.2 that lie far below the reference with mean offsets below -0.5 levels are the five ELLIPSE traits other than cohesion; FED-Dialogue depth, error_recovery, informative, and inquisitive; FED-Turn interesting; and USR-PC engaging.

“The 11 criteria of Section 5.2 that lie far below the reference with mean offsets below -0.5 levels are the five ELLIPSE traits other than cohesion; FED-Dialogue depth, error_recovery, informative, and inquisitive; FED-Turn interesting; and USR-PC engaging.”

On the unit aggregate's rank correlation with the holistic score, Jev Score has 0.597.

“and Jev Score 0.597.”

The eight HelpSteer2 correctness pairs on which Jev Choice abstains are mostly responses with little checkable factual content.

“The eight HelpSteer2 correctness pairs are mostly responses with little checkable factual content: two answer a request for”

No judge chose CANNOT_ASSESS on the binary panels.

“On the binary panels the option was CANNOT_ASSESS, whose definition in Table A2 describes the evidence available to the judge and asks for rare use, and no judge chose it there (Appendix D).”

The difference in these verdicts changes a panel's binary accuracy only on USR-PC.

“These verdicts differ on 4 of the 364 pairs and change a panel's binary accuracy only on USR-PC (Table C3f).”

• cites Table C3fNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Nearly all of the FED-Dialogue graded abstentions are on error_recovery.

“Most of each LLM judge's graded abstentions are on FED-Dialogue, nearly all of those on error_recovery, and most of them fall on pairs where Jev Choice also abstained (Table F1 gives the Jaccard indices).”

Against the same leave-one-out targets, the judges reach 73.0% (bare Noul) to 84.1% (Jev Choice), which is 3.2 to 14.3 points below the reference.

“Against the same leave-one-out targets the judges reach 73.0% (bare Noul) to 84.1% (Jev Choice), which is 3.2 to 14.3 points below the reference.”

Jev Choice's mean probability on the NA option is below 0.02 on each of the other 32 graded criteria.

“Its mean probability on the NA option is 0.57 on error_recovery, 0.11 on HelpSteer2 correctness, 0.023 on LFQA factuality, and below 0.02 on each of the other 32 graded criteria.”

Jev Score's probabilities give 66.7% exact accuracy when read by its own expected level.

“and 66.7% when read by its own expected level.”

Jev Choice's probabilities read by argmax, as Jev Choice runs, give 56.2% exact accuracy.

“Read by argmax, as Jev Choice runs, they give 56.2%.”

When Jev Choice and Jev Score differ on ELLIPSE, they never differ by more than one level.

“Jev Choice and Jev Score give different levels on 215 of ELLIPSE's 1,548 pairs and never differ by more than one level.”

Jev Score's level is the higher one on 179 of the 215 disagreements.

“Jev Score's level is the higher one on 179 of the 215.”

Because abstentions leave the denominator, Jev Choice's FED-Dialogue exact accuracy rests on 208 ordinal pairs.

“Because abstentions leave the denominator (Section 3.5), Jev Choice's FED-Dialogue exact accuracy rests on 208 ordinal pairs and Jev Score's on 225.”

The two rounding rules give different levels on 12 of Jev Score's 27 exact halves.

“Of Jev Score's 3,414 answers on the graded panels, 27 are exact halves, and the two rules give different levels on 12 of them.”

Because abstentions leave the denominator, Jev Score's FED-Dialogue exact accuracy rests on 225 ordinal pairs.

“Because abstentions leave the denominator (Section 3.5), Jev Choice's FED-Dialogue exact accuracy rests on 208 ordinal pairs and Jev Score's on 225.”

Jev Choice and Jev Score give different levels on 215 of ELLIPSE's 1,548 pairs.

“Jev Choice and Jev Score give different levels on 215 of ELLIPSE's 1,548 pairs and never differ by more than one level.”

Averaged over pairs, Jev Choice puts less probability than Jev Score on 'Somewhat.' (0.171 against 0.210).

“and less on”

Of Jev Score's 3,414 answers on the graded panels, 27 are exact halves.

“Of Jev Score's 3,414 answers on the graded panels, 27 are exact halves, and the two rules give different levels on 12 of them.”

For FED-Turn fluent, the best shift raises the reference from 70.9% to 87.2%.

“For FED-Turn fluent it raises the reference from 70.9% to 87.2%.”

None of the four other gap measures correlates significantly with the mean offset.

“None of the four other measures correlates significantly with the mean offset.”

Under the argmax rule, exchanging the probabilities accounts for 80% of the 10.5-point difference between the framings as run.

“Under the argmax rule, exchanging the probabilities accounts for 80% of the 10.5-point difference between the framings as run.”

Jev Choice's abstentions fall mostly on pairs where the NA option's second reading (criterion does not apply) fits.

“Jev Choice's abstentions fall mostly on pairs where that second reading fits.”

Jev Choice, Gemini, and DeepSeek use the middle level least.

“Jev Choice, Gemini, and DeepSeek use the middle level least.”

Within panels, criterion length shows no consistent relation with Jev's per-criterion accuracy.

“Within panels, criterion length shows no consistent relation with Jev's per-criterion accuracy”

The upper median is the most accurate of the three juries on every panel except USR-TC.

“The upper median is the most accurate of the three juries on every panel except USR-TC.”

For the other judges, the share of ELLIPSE errors below the label lies between 98.4% and 99.5%, except for DeepSeek at 96.9%.

“For the other judges it lies between 98.4% and 99.5%, except for DeepSeek at 96.9%.”

Of Jev Choice's 117 USR-TC errors, 17 are understandable pairs labelled UNMET.

“and 17 are understandable pairs labelled UNMET.”

The remaining 21 errors are the only errors that fall neither on a split pair nor under a candidate convention.

“They are the only errors that fall neither on a split pair nor under a candidate convention”

The raters marked self-contradictory responses such as the one quoted UNMET.

“The raters nonetheless marked selfcontradictory responses such as the one quoted UNMET”

The ELLIPSE example of Table H2 is one of Jev Choice's confident errors.

“The ELLIPSE example of Table H2 is one of Jev Choice's confident errors (Appendix J).”

• cites Table H2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Appendix JNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Nearly every ELLIPSE error lies below the label.

“Nearly every ELLIPSE error lies below the label.”

The criterion-level AUROCs show confidence fails inside a panel: AUROC is below 0.45 on five criteria with at least five pairs in each outcome: ELLIPSE phraseology and conventions, FED-Turn interesting, FED-Dialogue informative, and LFQA factuality.

“The AUROC is below 0.45 on five criteria with at least five pairs in each outcome: ELLIPSE phraseology and conventions, FED-Turn interesting, FED-Dialogue informative, and LFQA factuality.”

For Jev Choice's errors the shares are 58.4% and 27.1%.

“For Jev Choice's errors the shares are 58.4% and 27.1%.”

• AssumptionRests on an assumption: The three LLM judges' verdicts are independent under the independence baseline.

Measured from the two-rating mean before rounding, every offset is smaller in magnitude by 0.19 to 0.27 levels.

“Measured from the two-rating mean before rounding, every offset is smaller in magnitude, by 0.19 to 0.27 levels”

Among crowd-rated panels, Jev Choice's gap runs from +13.3 points on LFQA to -16.3 on FED-Dialogue.

“among the crowd-rated panels the gap runs from +13.3 points on LFQA to -16.3 on FED-Dialogue (Table H3).”

Three of the twelve sampled USR-TC contexts carry the literal string _nofact in place of a fact.

“Three of the twelve sampled USR-TC contexts carry the literal string _nofact in place of a fact”

It moves no other band of Figure 4b by more than 4.0 points.

“It moves no other band of Figure 4b by more than 4.0 points.”

For Jev Choice, the confidence field matches the general form (K p_max -1)/(K -1), where p_max is the largest returned probability and K counts every option including NA.

“For Jev Choice the field matches the general form (Kp max -1)/(K -1), where p max is the largest returned probability and K counts every option, the NA option included.”

HelpSteer2 shows a related pattern at a level that is neither the top nor absolute.

“HelpSteer2 shows a related pattern at a level that is neither the top nor absolute.”

The judges that trail the no-read baseline significantly do so with p ≤ 2.1 × 10 -4 in pair-level sign tests.

“The judges that trail it significantly (Section 4.1) do so with p ≤ 2.1 × 10 -4 in pair-level sign tests.”

• cites Section 4.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

None of the five judges chose complexity level 4 or 5 in 450 predictions.

“None of the five judges chose complexity level 4 or 5 in 450 predictions”

LFQA factuality concentrates its panel's errors: Jev Choice is wrong on 59.3% of its 118 pairs, against 18.3% and 16.7% on the panel's other two criteria.

“Jev Choice is wrong on 59.3% of its 118 pairs, against 18.3% and 16.7% on the panel's other two criteria.”

The share of ELLIPSE errors below the label is 99.5% for Jev Choice.

“The share is 99.5% for Jev Choice.”

The rater reference's own best shift raises it on PersonaChat from 61.1% to 80.0%.

“The rater reference's own best shift raises it there from 61.1% to 80.0%.”

The other four judges fall 1.7 (Luna) to 10.5 points (Jev Choice) short of 80.0% on PersonaChat.

“the other four judges fall 1.7 (Luna) to 10.5 points (Jev Choice) short of it.”

Outside ELLIPSE, Jev Choice's error rate falls between the 0.25-0.5 band and the top band on every panel, by 13.3 to 39.6 points.

“Outside ELLIPSE, Jev Choice's error rate falls between the 0.25-0.5 band and the top band on every panel, by 13.3 to 39.6 points (Table I1a)”

Jev Choice makes 117 errors on USR-TC, counting its Jev Noul verdicts on the two binary criteria.

“Jev Choice makes 117 errors on USR-TC, counting its Jev Noul verdicts on the two binary criteria.”

Restricted to ordinal pairs, the three-judge median is above that judge only on USR-TC, by 1.4 points (interval -2.8 to +5.1).

“Restricted to ordinal pairs, the three-judge median is above that judge only on USR-TC, by 1.4 points (interval -2.8 to +5.1).”

Over all 92 of Jev Noul's HealthBench errors the three LLM judges repeat 55.8% of their verdicts, against a baseline of 27.5%.

“Over all 92 of Jev Noul's HealthBench errors the three LLM judges repeat 55.8% of their verdicts, against a baseline of 27.5%.”

• AssumptionRests on an assumption: The three LLM judges' verdicts are independent under the independence baseline.

The four criteria that then stay more than 5 points short of the reference are FED-Dialogue informative and diverse, LFQA factuality, and USR-PC engaging.

“The four criteria that then stay more than 5 points short of the reference are FED-Dialogue informative and diverse, LFQA factuality, and USR-PC engaging.”

The same regression of one rater's level on the other's, over 3,096 ordered pairs of ratings, gives 0.50.

“The same regression of one rater's level on the other's, over 3,096 ordered pairs of ratings, gives 0.50.”

One rater is within one level of the other on 98.2% of ratings.

“one rater is within one level of the other on 98.2% of ratings.”

All five of those criteria carry a negative offset for Jev Choice, from -0.58 levels on factuality to -1.38 on phraseology.

“All five carry a negative offset for Jev Choice, from -0.58 levels on factuality to -1.38 on phraseology.”

With each panel's best LLM judge as fallback, cross-fitted gains run from -2.1 to +1.3 points.

“With each panel's best LLM judge as fallback they run from -2.1 to +1.3 points.”

On Topical-Chat no shift raises the rater reference or any matched judge.

“On Topical-Chat no shift raises the rater reference or any matched judge.”

Gemini's shifted accuracy on PersonaChat equals 80.0%.

“Gemini's shifted accuracy equals that value”

On two panels a no-read baseline is competitive on exact accuracy.

“On two panels a no-read baseline (Section 3.4) is competitive on exact accuracy.”

• cites Section 3.4Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

HealthBench's confident errors are Jev Noul's six false positives with highest probability of yes, from 0.91 to 0.96, and six false negatives with lowest, from 0.08 to 0.12.

“They are Jev Noul's six false positives with the highest probability of yes, from 0.91 to 0.96, and the six false negatives with the lowest, from 0.08 to 0.12.”

Every judge is correct on at most 34.8% of the 23 pairs labelled UNMET.

“every judge is correct on at most 34.8% of the 23 pairs labelled UNMET.”

Counting LLM abstentions and call failures as non-repeats lowers no band by more than 1.9 points.

“Counting LLM abstentions and call failures as non-repeats lowers no band by more than 1.9 points.”

Of Jev Choice's 117 USR-TC errors, 16 are the _nofact pairs of uses_knowledge.

“Of these, 16 are the _nofact pairs of uses_knowledge”

The formula does not describe Jev Score's confidence (median deviation 0.10, maximum 0.46), which is used as returned.

“The formula does not describe Jev Score's confidence (median deviation 0.10, maximum 0.46), which is used as returned.”

The length of ELLIPSE's six criterion sentences orders Jev Choice's per-trait accuracy perfectly (Spearman 1.00).

“yet their length orders Jev Choice's per-trait accuracy perfectly (Spearman 1.00)”

On FED-Dialogue, Jev Choice's accuracy is lowest in the shortest quartile (37.5%).

“and on FED-Dialogue it is lowest in the shortest quartile (37.5%).”

Two variants leave the rise with confidence intact.

“Two variants leave the rise with confidence intact.”

The three largest falls in Jev Choice's error rate between confidence bands are on FED-Turn, LFQA, and USR-TC.

“and the three largest falls are on FED-Turn, LFQA, and USR-TC.”

By quartile of unit length, Jev Choice's accuracy on ELLIPSE falls from 16.4% in the shortest quartile to 11.2-12.3% in the two longest.

“By quartile of unit length, Jev Choice's accuracy on ELLIPSE falls from 16.4% in the shortest quartile to 11.2-12.3% in the two longest”

Apart from ELLIPSE, the share of ordinal pairs on which all four matched judges are wrong is 17.2% to 31.5%.

“Apart from ELLIPSE (Section 5.1), it is 17.2% to 31.5%.”

Pooled over the six traits, individual ratings put 25.4% of ratings at levels 4-5, fewer than the labels' 36.3%.

“Pooled over the six traits, individual ratings put 25.4% of ratings at levels 4-5, fewer than the labels' 36.3%”

Jev Score is the best single judge in 99 of the 100 FED-Turn halves.

“Jev Score is the best single judge in 99 of the 100 FED-Turn halves.”

For labels of level 3 to level 5, Jev Choice most often predicts level 2.

“For labels of level 3 to level 5, Jev Choice most often predicts level 2”

Scale width changes the chance level of exact accuracy.

“Scale width changes the chance level of exact accuracy (Section 3.5)”

Across panels, the three with the shortest units (FED-Turn, USR-PC, USR-TC) have gaps from 16.2 points below the reference to 0.3 points above it.

“Across panels, the three with the shortest units, FED-Turn, USR-PC, and USR-TC, have gaps that run from 16.2 points below the reference to 0.3 points above it.”

Of Jev Choice's 117 USR-TC errors, another 63 are ordinal errors on split pairs.

“Another 63 are ordinal errors on split pairs.”

No judge places more than 9.3% of any trait's predictions at levels 4-5.

“no judge places more than 9.3% of any trait's predictions there (Table G1c).”

• cites Table G1cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Panels with the same scale can still differ widely in Jev Choice's gap.

“but panels with the same scale can still differ widely.”

Jev Choice's within-one accuracy is 64.3%.

“Jev Choice's within-one accuracy is 64.3%”

In pair-level sign tests, every judge's p against the provenance baseline is at least 0.27.

“In the pair-level sign tests of Section 4.1, every judge's p against the provenance baseline is at least 0.27.”

• cites Section 4.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Under the independence baseline the three LLM judges would repeat 126.7 of their 252 verdicts.

“Under the independence baseline the three would repeat 126.7 of their 252 verdicts.”

• AssumptionRests on an assumption: The three LLM judges' verdicts are independent under the independence baseline.

On PersonaChat every shifted judge exceeds the unshifted rater reference.

“On PersonaChat every shifted judge exceeds the unshifted rater reference”

Its best result, on USR-PC, is 1.0 point above the most accurate matched judge, with an interval of -2.3 to +4.3 points.

“Its best result, on USR-PC, is 1.0 point above the most accurate matched judge, with an interval of -2.3 to +4.3 points.”

The ELLIPSE offset grows with the holistic band.

“The ELLIPSE offset also grows with the holistic band (Appendix G)”

The LLM judges' explanations show that they read the string _nofact as the absence of a fact.

“The LLM judges' explanations show that they read the string as the absence of a fact (Table H2).”

• cites Table H2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

ELLIPSE, the only graded panel with trained raters, has Jev Choice's largest gap.

“ELLIPSE, the only graded panel with trained raters, has Jev Choice's largest gap (Section 5.5)”

On the six panels whose oracle cost in Table 4 is 10% to 59%, the cross-fitted cascade matches the best LLM judge on the held-out half in only 46% to 64% of halves.

“On the six panels whose oracle cost in Table 4 is 10% to 59%, the crossfitted cascade matches the best LLM judge on the held-out half in only 46% to 64% of halves.”

The fallback leads Jev on the deferred pairs of every panel.

“The fallback leads Jev on the deferred pairs of every panel.”

Routing any one or two of LFQA's three criteria to Luna gives at most 70.9%, reached with factuality and amount_info together.

“Routing any one or two of the three criteria to Luna gives at most 70.9%, reached with factuality and amount_info together.”

HealthBench's 2.0-point gain over Gemini rests on a kept-pair lead of 18 to 10 (p = 0.18).

“HealthBench's 2.0-point gain over Gemini rests on a kept-pair lead of 18 to 10 (p = 0.18).”

Where Jev alone is at least as accurate as a fallback, as for all three RiceChem fallbacks, the reach is zero.

“Where Jev alone is at least as accurate as a fallback, as for all three RiceChem fallbacks, the reach is zero.”

The fixed threshold τ = 0.75 falls below Gemini alone on ELLIPSE and FED-Dialogue.

“The fixed threshold falls below Gemini alone on ELLIPSE and FED-Dialogue.”

RiceChem is the one panel where Jev leads significantly on its kept pairs (63 to 26, p = 1 × 10 -4 ).

“RiceChem is the one panel where Jev leads significantly on its kept pairs (63 to 26, p = 1 × 10 -4 ).”

Correlated errors undo the cascade advantage.

“Correlated errors undo that advantage.”

These cost and time ratios understate Jev's advantage at equal concurrency.

“These ratios understate Jev's advantage at equal concurrency.”

Both separations on HelpSteer2 rank judges that trail a constant predictor.

“Both separations on that panel, Jev Choice's lead over Luna and Gemini's over Jev Choice, therefore rank judges that trail a constant.”

The cost ratio tends to grow with the number of criteria per unit, because each criterion adds an LLM call but no Jev request.

“The cost ratio tends to grow with the number of criteria per unit, because each criterion adds an LLM call but no Jev request”

Gemini's per-token price and its reasoning tokens widen its cost ratio further.

“Gemini's per-token price and its reasoning tokens widen its cost ratio further.”

The judges received those same descriptions, so any population norm reached the raters through their training.

“The judges received those same descriptions, so any population norm reached the raters through their training.”

Label noise alone does not explain the errors on ELLIPSE, LFQA factuality, or USR-TC uses_knowledge.

“Label noise alone does not explain the errors on ELLIPSE, LFQA factuality, or USR-TC uses_knowledge.”

What limits the cascade on ELLIPSE and FED-Dialogue is Jev's accuracy.

“What limits the cascade there is Jev's accuracy.”

A reluctance to grant high levels, shared by all judges, is harder to exclude.

“A reluctance to grant high levels, shared by all judges, is harder to exclude.”

The PersonaChat labels are corpus-relative.

“We read the PersonaChat labels as corpus-relative.”

The panel with the largest gap, ELLIPSE, is the one rated by trained raters, which is weak evidence against an account based on label quality.

“The panel with the largest gap, ELLIPSE, is the one rated by trained raters, which is weak evidence against an account based on label quality.”

The comparison favours the judges where most labels are at the top of a three-level scale, because a clipped shift of one level then partly amounts to predicting the top level and would raise a rater's agreement too.

“The comparison favours the judges where most labels are at the top of a three-level scale, because a clipped shift of one level then partly amounts to predicting the top level and would raise a rater's agreement too.”

Most of the correlation between mean offset and mean gap is mechanical, because a large offset forces low exact accuracy.

“Most of this correlation is mechanical, because a large offset forces low exact accuracy.”

Because the 5-point equivalence margin is absolute, it is lenient where accuracy is low, as on ELLIPSE.

“The 5-point equivalence margin is post hoc (Section 3.5), and because it is absolute it is lenient where accuracy is low, as on ELLIPSE.”

With six traits the level descriptions cannot be distinguished from other trait differences, such as criterion length.

“with six traits the level descriptions cannot be distinguished from other trait differences, such as criterion length (Appendix H).”

On a split pair, part of what counts as a repeated error may be disagreement among the raters.

“On a split pair, part of what counts as a repeated error may be disagreement among the raters.”

ELLIPSE's labels carry a population norm.

“We read ELLIPSE's labels as carrying a population norm.”

Where a level's meaning depends on a population, a corpus, or an annotation habit, every judge falls short in the same direction, the best LLM judge included.

“Where, on our reading, a level's meaning depends on a population, a corpus, or an annotation habit, every judge falls short in the same direction, the best LLM judge included.”

The offset leaves the order of the essays largely intact.

“The offset leaves the order of the essays largely intact.”

Part of that rise is mechanical, because 63.3% of the PersonaChat labels are at the top level.

“Part of that rise is mechanical, because 63.3% of the PersonaChat labels are at the top level.”

Because the graded panels also change the domain and the options and instruction that Jev receives, the paper attributes no failure to scale type alone.

“Because the graded panels also change the domain and the options and instruction that Jev receives, we attribute no failure to scale type alone.”

The rounding rule does not produce the sign of the offset.

“so the rounding rule does not produce the sign of the offset”

The wrapper's gain comes from false positives.

“The gain comes from false positives.”

Because each criterion holds only 4 to 25 pairs, these differences show only that a fixed wrapper can lower accuracy on individual criteria.

“Because each criterion holds only 4 to 25 pairs, these differences show only that a fixed wrapper can lower accuracy on individual criteria.”

Label uncertainty of this kind can distort any comparison with human agreement.

“Label uncertainty of this kind can distort any comparison with human agreement (Elangovan et al., 2025).”

• cites Elangovan et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Cascade gains under oracle thresholds are optimistic.

“Cascade gains under oracle thresholds are optimistic, the cross-fitted estimates rest on halves as small as 12 units, cascade costs are linear estimates, and only deferral on the confidence of Jev Choice or Jev Noul was tried.”

Unit length has little effect.

“Unit length has little effect.”

The Both shifted row, which shifts judges and raters alike, is the like-for-like comparison.

“The Both shifted row, which shifts judges and raters alike, is therefore the like-for-like comparison.”

• AssumptionRests on an assumption: Shifting both judges and raters alike yields a like-for-like comparison.

Score's expected level weights the level numbers by probabilities obtained from level descriptions evaluated one at a time.

“Score's expected level thus weights the level numbers by probabilities obtained from level descriptions evaluated one at a time.”

The error_recovery label rates recovery under a condition that the criterion text does not state.

“If so, the label rates recovery under a condition that the criterion text does not state.”

No run offering one wording on both kinds of panel was made.

“Offering one wording on both kinds of panel would separate the wording from these differences, and no such run was made.”

The numeric ratings on error_recovery likely come mainly from raters who saw an error to recover from.

“Because most N/A answers report that no system error occurred, the numeric ratings likely come mainly from raters who saw an error to recover from.”

• AssumptionRests on an assumption: Raters who saw no system error chose the N/A option, and raters who saw an error provided numeric ratings.

The framing that answers one level higher gains exact accuracy on its disagreements.

“Because every judge places ELLIPSE essays below their labels (Section 5.3), the framing that answers one level higher gains exact accuracy on its disagreements.”

The contrast between the two abstain options does not isolate the wording.

“The two options also differ in the instruction to use the option rarely, in scale type, and in panel, so this contrast does not isolate the wording.”

Over all 406 pairs, the physician reference gives 98.1%, an artifact of selection.

“Over all 406 pairs this gives 98.1%, an artifact of selection.”

The PersonaChat comparison favours the judges for the reason given in Section 5.3.

“a comparison that favours the judges for the reason given in Section 5.3.”

• cites Section 5.3Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The 0.50 slope is not a like-for-like benchmark for the judges, whose target, the two-rater label, is less noisy than a single rating.

“It is not a like-for-like benchmark for the judges, whose target, the two-rater label, is less noisy than a single rating.”

The judges compress the scale.

“The judges compress the scale.”

With six ELLIPSE traits, criterion length cannot be separated from what each trait's levels describe.

“With six traits, length cannot be separated from what each trait's levels describe (Table 3).”

The half-up rule raises each trait's labels by 0.19 to 0.27 levels.

“which is the amount by which the half-up rule raises each trait's labels.”

Rater expertise does not order the gaps.

“Rater expertise does not order the gaps either.”

With few criteria per panel the test of criterion length is weak.

“and with few criteria per panel the test is weak.”

Unit length has no common effect on Jev Choice's accuracy across panels.

“Unit length has no common effect.”

The four criteria that stay more than 5 points short of the reference are candidates for conventions that a shift of location does not capture, such as LFQA's absolute top level.

“They are the candidates for conventions that a shift of location does not capture, such as LFQA's absolute top level (Appendix H).”

A comparison with 0.50 understates the judges' compression.

“so a comparison with 0.50 understates the judges' compression.”

A judge with one rater's noise would show a slope above 0.50 against the label.

“A judge with one rater's noise would show a slope above 0.50 against the label”

Part of the rise in Figure 4b follows from each LLM judge's own tendencies given the criterion and the label.

“Part of the rise in Figure 4b therefore follows from each LLM judge's own tendencies given the criterion and the label.”

Because nearly all ELLIPSE errors share one sign, Jev Choice's mean absolute error and the magnitude of its mean signed error differ by less than 0.01 level.

“Because nearly all errors share one sign, Jev Choice's mean absolute error (1.256 levels) and the magnitude of its mean signed error (1.247 levels) differ by less than 0.01 level.”

The slope of 0.50 is below one because the two raters disagree.

“This slope is below one because the two raters disagree.”

The ELLIPSE slice does not separate essay length from essay quality.

“and this slice does not separate essay length from essay quality.”

Compression predicts the pattern that the offset grows with the label when predictions rise by less than one level per label level.

“Compression predicts this pattern.When predictions rise by less than one level per label level, the offset grows with the label.”

By Spearman correlation with the unrounded rater mean, Jev Choice ranks essays about as well as a single rater.

“By this measure Jev Choice ranks essays about as well as a single rater.”

Every offset stays negative because the downward rule's 0.47-level raise is less than the smallest offset magnitude, Gemini's 0.77 levels.

“That is less than the smallest offset magnitude, Gemini's 0.77 levels, so every offset stays negative.”