A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

May combine two claims.

A judge can be blind to one of scope or strength while catching the other; the paper finds that it is.

“The method may improve performance in this basin fails differently from the method improves performance in all basins and from the method substantially and consistently improves performance, and a judge can be blind to one while catching the other.We find that it is.”

May combine two claims.

Agreement with a validated label set is not evidence of validity; a construct-blind family reproduces 55%-67% of five public sets' labels.

“That agreement with a validated label set is evidence of validity, assumed nearly everywhere and the subject of §5.5, where a construct-blind family reproduces 55%-67% of five public sets' labels.”

May combine two claims.

Judges register that a claim now covers more and largely miss that it now commits harder.

“Judges register that a claim now covers more and largely miss that it now commits harder.”

• AssumptionRests on an assumption: For the constructs studied, the correct verdict y* is a fact about the relation between the claim and its evidence rather than about any annotator's opinion.
• AssumptionRests on an assumption: Human annotators can establish the direction of construct-changing interventions at agreement ≥ 0.75.
• AssumptionRests on an assumption: Drawing generation, verification, and judging from three disjoint model families prevents circularity.

May combine two claims.

Validation power, defined relative to an input mode, measured on 5 public sets in both modes: the same sets read as sound per item and as 55%-67% recoverable paired.

“Validation power, defined relative to an input mode (Definition 6, Corollary 2), and measured on 5 public sets in both modes.The mode is load-bearing: the same sets read as sound per item and as 55%-67% recoverable paired.”

• AssumptionRests on an assumption: A frozen family reading only surface form is a valid construct-blind predictor that captures surface form.

May combine two claims.

Across 5 public label sets, a frozen family reading only surface form, given the same paired input the judge gets, reproduces 55%-67% of their labels, including 67.4% of MT-Bench's human votes.

“Across 5 public label sets, a frozen family reading only surface form, given the same paired input the judge gets, reproduces 55%-67% of their labels, including 67.4% of MT-Bench's human votes.”

• AssumptionRests on an assumption: A frozen family reading only surface form is a valid construct-blind predictor that captures surface form.

May combine two claims.

RewardBench's length control: constraining chosen responses to be no longer than rejected ones did not remove the confound but inverted it.

“RewardBench's length control is the sharpest case: constraining chosen responses to be no longer than rejected ones did not remove the confound but inverted it, leaving prefer”

May combine two claims.

Measured on the released set, the chosen response is the longer one in 40.3% of pairs, so the content-free rule prefer the shorter response is correct on 59.7%.

“Measured on the released set, the chosen response is the longer one in 40.3% of pairs, so the content-free rule prefer the shorter response is correct on 59.7%.”

May combine two claims.

The two that manipulate length, verbosity_ padding and elaboration, fall below the bar at 0.689 and 0.404, while the other 3 invariance slots agree at ≥ 0.971.

“The two that manipulate length, verbosity_ padding and elaboration, fall below the bar at 0.689 and 0.404, while the other 3 invariance slots agree at ≥ 0.971.”

May combine two claims.

A surface-only paired predictor reproduces 67.4% of MT-Bench labels, the highest of the public sets and higher than either pair-role benchmark.

“Its labels are human pairwise votes, and a surface-only paired predictor reproduces 67.4% of them, the highest of the public sets and higher than either pair-role benchmark.”

• AssumptionRests on an assumption: MT-Bench labels are human pairwise votes.

May combine two claims.

Our corpus is the opposite, with its preferred side in a narrow absolute band, so both modes succeed.

“Our corpus is the opposite, with its preferred side in a narrow absolute band, so both modes succeed.”

May combine two claims.

Afterwards one annotator still called an appositive gloss directional (32 of 46) while the other two almost never did (3), and that annotator agreed with the others at 0.877 on the main batch.

“afterwards one annotator still called an appositive gloss directional (32 of 46) while the other two almost never did (3), and that annotator agreed with the others at 0.877 on the main batch.”

May combine two claims.

The signed change differs: scope edits lengthen (+4.1% median, 64% longer) and strength edits shorten (-2.5%, 39%).

“The signed change does differ, with scope edits lengthening (+4.1% median, 64% longer) and strength edits shortening (-2.5%, 39%).”

May combine two claims.

Of 20 ordered set pairs, only 2 admit transfer evaluation; the remaining 18 have label spaces that do not share two categories, so a predictor fitted on one cannot be scored on the other even in principle.

“Of 20 ordered set pairs, 2 admit the question at all: the remaining 18 have label spaces that do not share two categories, so a predictor fitted on one cannot be scored on the other even in principle.”

May combine two claims.

Whether naming an entity more precisely changes what a sentence claims is a question our codebook never answered, and one competent reader in three answers it the other way.

“Whether naming an entity more precisely changes what a sentence claims is a question our codebook never answered, and one competent reader in three answers it the other way.”

May combine two claims.

The slot distributions overlap across the axes: condition_removal is a strength slot detected at 0.379, above 2 of the four scope slots.

“condition_removal is a strength slot detected at 0.379, above 2 of the four scope slots.”

May combine two claims.

Every axis clears the bar: scope 0.898 (n = 144), strength 0.889 (n = 144), invariance 0.789 (n = 190).

“every axis clears the bar: scope 0.898 (n = 144), strength 0.889 (n = 144), invariance 0.789 (n = 190).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Pipeline discard before annotation was 24.2% overall, dominated by the length tolerance.

“Pipeline discard before annotation was 24.2% overall, dominated by the length tolerance”

May combine two claims.

The gap has the same sign on 7 of the 7 judges, spanning vendors, parameter scales and reasoning-enabled or not.

“The gap has the same sign on 7 of the 7 judges, spanning vendors, parameter scales and reasoning-enabled or not, with per-judge values in Table 5.”

May combine two claims. The quote only partly states this claim.

It clears both (the gap is above 0.10 and passes the paired bootstrap).

“It clears both.”

May combine two claims.

Permuting the axis label within the construct arm while holding every judge's grades fixed gives p < 0.002, with a largest null gap of +0.123.

“Permuting the axis label within the construct arm while holding every judge's grades fixed gives p < 0.002, with a largest null gap of +0.123.”

May combine two claims.

The first was ours: the generator padded sentences with as is often observed, in practice and as is well known, which read as filler but each assert something the evidence never stated.

“The first was ours: the generator padded sentences with as is often observed, in practice and as is well known, which read as filler but each assert something the evidence never stated.”

May combine two claims.

Three revisions of the generation prompt failed to stop it, and what worked was a fixed marker list checked programmatically before the verifier sees the pair (epistemic_marker_change): a prompt states an intention, a filter enforces a property.

“Three revisions of the generation prompt failed to stop it, and what worked was a fixed marker list checked programmatically before the verifier sees the pair (epistemic_marker_change): a prompt states an intention, a filter enforces a property.”

May combine two claims.

Figure 5 shows the vertical gap the study was built to look for, and it is large.

“Figure 5 shows the vertical gap the study was built to look for, and it is large.”

May combine two claims.

246 of 288 construct items carry a majority direction matching the intended one (116 scope, 130 strength) and 96 of 190 invariance items survive.

“246 of 288 construct items carry a majority direction matching the intended one (116 scope, 130 strength) and 96 of 190 invariance items survive”

May combine two claims.

On the 217 items where all 3 annotators agreed on direction, the pre-committed robustness split, the gap is +0.125: larger than on the full set.

“And on the 217 items where all 3 annotators agreed on direction, the pre-committed robustness split, the gap is +0.125: larger than on the full set, which is the direction that makes it a fact about judges rather than about annotation difficulty.”

May combine two claims.

Run the identical family, folds and nulls against a set whose labels are a function of the generative condition, and the paired predictor is correct on 99.6% of pairs against 55%-67% for the public sets.

“Run the identical family, folds and nulls against a set whose labels are a function of the generative condition, and the paired predictor is correct on 99.6% of pairs against 55%-67% for the public sets [6].”

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

After human filtering, the magnitude match between scope and strength edits survives, with median |∆| length 5.0% on scope and 5.3% on strength.

“The magnitude match survives human filtering.”

• Also stated in: § 5.2 The scope/strength asymmetry · plain

  “median |∆| length is 5.0% on scope against 5.3% on strength”

May combine two claims.

The strongest instrument in the sweep, anthropic/claude-haiku-4.5, reaches R = 0.561 and so still misses more than two construct changes in five while holding its verdict on 0.945 of edits that should not move it.

“The strongest instrument in the sweep, anthropic/claude-haiku-4.5, reaches R = 0.561 and so still misses more than two construct changes in five while holding its verdict on 0.945 of edits that should not move it.”

May combine two claims.

Scope is nominally higher, but by less than the spread between annotator pairs, so we claim no ordering.

“scope is nominally higher, but by less than the spread between annotator pairs, so we claim no ordering”

May combine two claims.

Quantifier_strengthening and domain_extension are the most detected at R=0.445, and hedging_removal the least at R=0.210.

“quantifier_strengthening and domain_extension are the most detected at 0.445, and hedging_removal the least at 0.210”

May combine two claims.

D in practice is the empirical distribution over base items that survived Protocol 1, which is not the distribution over items a judge meets in deployment.

“D in practice is the empirical distribution over base items that survived Protocol 1, which is not the distribution over items a judge meets in deployment.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both typography normalisation and multi-span acceptance cost the authors a batch of correct labels before they were understood.

“Both cost us a batch of correct labels before they were understood.”

May combine two claims.

Measured as a profile over 7 judges and 4 domains with direction set by 3 humans at agreement 0.852, judges sit at S = 0.945 against R = 0.319.

“Measured as a profile over 7 judges and 4 domains with direction set by 3 humans at agreement 0.852, judges sit at S = 0.945 against R = 0.319”

• AssumptionThe authors state this as an assumption: A T_C edit really changes the correct verdict.

May combine two claims.

A judge can offer many usable cuts while reproducing its own grade on the same item a small fraction of the time at temperature 0.

“another can offer many usable cuts while reproducing its own grade on the same item a small fraction of the time at temperature 0”

May combine two claims.

In the measured roster, usable operating points span 2 to 9 and regrade agreement 0.412 to 1.000 (Table 5).

“Both appear in the measured roster, where usable operating points span 2 to 9 and regrade agreement 0.412 to 1.000 (Table 5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paradox result is correlational.

“The paradox result is correlational”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The taxonomy was internally contradictory and still looked fine as a list of names.

“The taxonomy was internally contradictory and still looked fine as a list of names”

Classified as probably not a substantial claim.

Three boundaries carry most of the disagreement.

“Three boundaries carry most of the disagreement.”

Classified as probably not a substantial claim. May combine two claims.

Source sentences carry non-breaking hyphens, en dashes and curly quotes; an annotator who retypes produces ASCII forms; an unnormalised substring test rejects a correct label over one character.

“source sentences carry non-breaking hyphens, en dashes and curly quotes, an annotator who retypes produces the ASCII forms, and an unnormalised substring test rejects a correct label over one character.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of 478 annotated pairs, 246 construct and 96 control items survive, 2 carry no label and 2 slots were excluded.

“Of 478 annotated pairs, 246 construct and 96 control items survive, 2 carry no label and 2 slots were excluded; per-slot yields are in Table4.”

May combine two claims.

The scope-strength gap is positive on 7/7 judges, ranging from +0.003 to +0.222.

“scope -strength gap: positive on 7/7 judges, +0.003 to +0.222”

May combine two claims.

tense_modality_shift has ordinary agreement and low yield.

“tense_modality_shift has ordinary agreement and low yield -readers agreeing that the edit changed nothing”

May combine two claims.

An earlier build gated the construct arm on the verifier agreeing that an edit widened the claim, and that gate rejected 12 of 12 verified strength edits.

“An earlier build gated the construct arm on the verifier agreeing that an edit widened the claim, and that gate rejected 12 of 12 verified strength edits”

May combine two claims.

Neither usable points nor regrade predicts the other.

“Rows are sorted by usable points and the regrade column does not follow that order, so neither quantity predicts the other.”

May combine two claims.

The two length-manipulating invariance slots have ordinary yield and agreement below the pre-set bar of 0.75.

“while the two lengthmanipulating invariance slots have ordinary yield and agreement below the pre-set bar of 0.75.”

May combine two claims.

Judges disagree with humans more than raw agreement suggests, disagree with themselves across reorderings, and move on properties that carry no information about quality such as response length, formatting, order, and injected metadata.

“Judges disagree with humans more than raw agreement suggests, they disagree with themselves across reorderings, and they move on properties that carry no information about quality: response length [60], list and emphasis formatting [59], the order the candidates are shown in [54], injected metadata about the author [33].”

• cites [60]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [59]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [54]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Membership in intervention classes depends on the criterion c: register normalisation is construct-preserving when the criterion is claim calibration and construct-changing when the criterion is tone.

“Membership depends on c: register normalisation is in T I when the criterion is claim calibration and in T C when the criterion is tone.”

May combine two claims.

The leak is not in the judge but in the label set: when a label is inherited from the generative condition that produced an item, and that condition also fixes surface form, the set is solvable without the construct, and agreement with it rewards provenance rather than the construct.

“The leak is not in the judge but in the label set: when a label is inherited from the generative condition that produced an item, and that condition also fixes surface form, the set is solvable without the construct, and agreement with it rewards provenance rather than the construct.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A valid judge must satisfy two conditions: under a construct-preserving edit the verdict must not change, and under a minimal construct-changing edit it must change.

“A valid judge must satisfy two conditions.Under an edit that preserves the construct, the verdict must not change.Under a minimal edit that changes the construct, it must.”

May combine two claims.

Prior judge evaluation measures invariance (verdict stability under construct-preserving edits) and never measures construct sensitivity (verdict change under construct-changing edits).

“Prior judge evaluation measures the first and, to our knowledge, never the second.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A demand for accuracy is a demand to sound certain, which narrows scope while amplifying strength, because hedges and stated conditions read as uncertainty.

“On two it is expected: a demand for accuracy is a demand to sound certain, which narrows scope while amplifying strength, because hedges and stated conditions are precisely what reads as uncertainty.”

May combine two claims.

Membership in intervention classes is unrelated to edit magnitude: a two-word quantifier change is construct-changing while a rewritten paragraph can be construct-preserving, so the classes cannot be approximated by an edit-distance threshold.

“Membership is unrelated to edit magnitude: a two-word quantifier change is in T C while a rewritten paragraph can be in T I , so the classes cannot be approximated by an edit-distance threshold.”

Classified as probably not a substantial claim.

A large judge audit reports verbosity bias as negligible, under 0.011 across all 21 judges, over roughly 541,000 judgments.

“A large judge audit reports verbosity bias as negligible, under 0.011 across all 21 judges, over roughly 541,000 judgments [38].”

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Reliability and validity come apart; a judge can be stable and unbiased on surface properties yet fail to notice when the construct has changed.

“Reliability and validity come apart, and a judge can be stable, unbiased on every surface property yet tested, and still fail to notice when the thing it is supposed to measure has changed.”

May combine two claims.

Membership in intervention classes is not observable from x alone; it is a claim about y*, which is why a pipeline that assigns it by model output is circular.

“And membership is not observable from x alone; it is a claim about y ⋆ , which is why Protocol 1 exists and why a pipeline that assigns it by model output is circular.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Definition 2 is not new as a form: it is the invariance / directional-expectation pair of Ribeiro et al., stated for an evaluator rather than for a task model.

“Definition 2 is not new as a form: it is the invariance / directional-expectation pair of Ribeiro et al. [45], stated for an evaluator rather than for a task model.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

What has been missing on the evaluator side is not the vocabulary but an operational test.

“What has been missing on the evaluator side is not the vocabulary but an operational test, and the rest of this section is one.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Construct validity is the question of whether an instrument measures the theoretical construct it is claimed to measure, as opposed to correlating with it; modern treatment makes validity a property of an interpretation of scores; measurement-theoretic reading insists validity is a claim about a causal relation between attribute and score.

“Construct validity is the question of whether an instrument measures the theoretical construct it is claimed to measure, as opposed to correlating with it [10]; the modern treatment makes validity a property of an interpretation of scores rather than of the instrument in isolation [35], and the measurement-theoretic reading insists that an instrument's validity is a claim about a causal relation”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The invariance / directional-expectation pair is Ribeiro et al.'s, named for task models; the paper claims its systematic application to evaluators.

“The invariance / directional-expectation pair is Ribeiro et al. [45]'s, named for task models; we claim its systematic application to evaluators.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The graded scale is part of the instrument; a judge that grades coarsely has a coarser frontier, and some measured insensitivity may be attributable to scale granularity rather than to the judge.

“What it costs is that the graded scale is part of the instrument now: a judge that grades coarsely has a coarser frontier, and Appendix H S3 reports how much of the measured insensitivity is attributable to scale granularity rather than to the judge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The two axes of T C were fixed in probes/edit_taxonomy.py before any judge was run.

“T C splits into two axes, fixed in probes/edit_taxonomy.py before any judge is run”

May combine two claims.

For every (s, r) in [0,1]^2 there exists a judge attaining V = (s, r); neither coordinate constrains the other, and no inequality between them holds in general.

“Proposition 1 (The coordinates are independent).For every (s, r) ∈ [0, 1] 2 there exists a judge attaining V = (s, r).In particular neither coordinate constrains the other, and no inequality between them holds in general.”

Classified as probably not a substantial claim.

Over-generalisation under that pressure is measured at scale.

“over-generalisation under that pressure is measured at scale [41]”

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The one quantity that would cross between the two submissions, a bound on judge accuracy from a per-sentence gold set, is unrun in both and stated as such in both.

“the one quantity that would cross between them, a bound on judge accuracy from a per-sentence gold set, is unrun in both and stated as such in both ( §5.5).”

Classified as probably not a substantial claim. May combine two claims.

Every profile is read off a frontier obtained by eliciting a graded verdict and cutting it ourselves, so all judges are compared on identical terms whatever their vendor exposes.

“Every profile is read off a frontier we obtain by eliciting a graded verdict and cutting it ourselves, so all judges are compared on identical terms whatever their vendor exposes.”

Classified as probably not a substantial claim. May combine two claims.

A judge exposing a decision threshold can trade one coordinate for the other, so a single profile is one point on a curve and two judges compared at their defaults are two arbitrary points on two different curves.

“A judge exposing a decision threshold can trade one coordinate for the other, so a single profile is one point on a curve and two judges compared at their defaults are two arbitrary points on two different curves.”

May combine two claims.

Commitment is representable, steerable and linearly decodable, and judges still do not act on it.

“commitment is representable, steerable and linearly decodable, and judges still do not act on it.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The axes are separable in the sense measurement requires: an edit on one can be applied while holding the other's operators fixed, and a verifier can check which moved.

“The axes are separable in the sense measurement requires: an edit on one can be applied while holding the other's operators fixed, and a verifier can check which moved.”

Probably not a substantial claim: may not pass S4 (based).

Every choice in the method section was frozen in the repository before the first judge call.

“Every choice here is frozen in the repository before the first judge call”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Scope and strength do not exhaust overreach; they are the two that can be edited minimally and verified by humans.

“scope and strength do not exhaust overreach; they are the two that can be edited minimally and verified by humans.”

Classified as probably not a substantial claim. May combine two claims.

The direction assignment protocol uses three disjoint model families, with humans holding the deciding vote.

“Three disjoint model families, and humans hold the deciding vote.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A benchmark can be solvable without doing the task is a general phenomenon, with examples including shortcut learning, syntactic heuristics, annotator identity recoverable, counterfactually augmented data, benchmarking critique, and reward hacking.

“That a benchmark can be solvable without doing the task is a general phenomenon: shortcut learning in deep networks [17], syntactic heuristics passing inference benchmarks [34], annotator identity recoverable from labels [18], counterfactually augmented data proposed against such correlations [28], the benchmarking critique that a leaderboard number can be uninformative about the capability it”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Jiang et al. formalise a factuality/specificity trade-off and rewrite claims to trade one against the other; they ask how a generator should set specificity, while this paper asks whether an evaluator notices when specificity has been changed for it.

“Jiang et al. [27] formalise a factuality/specificity trade-off and rewrite claims to trade one against the other.The distinction is the direction of the question: they ask how a generator should set the specificity of what it writes, we ask whether an evaluator notices when specificity has been changed for it.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Behavioural testing supplies the invariance versus directional expectation distinction; judge evaluation took up the first and not the second, so the paper's claim is the application to evaluators, not the distinction.

“Behavioural testing supplies the invariance versus directional expectation distinction we adopt [45]; judge evaluation took up the first and not the second, so what we claim is the application to evaluators, not the distinction.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Both coordinates are individually gameable in opposite directions: a constant judge gives V=(1,0) and a flip judge gives V=(0,1); neither reads x in a way that depends on c.

“Proposition 2 (Both coordinates are individually gameable, in opposite directions).Let J const (x, c) = y 0 for a fixed y 0 , and let J flip return a verdict drawn to differ from the base verdict whenever the input was produced by any intervention.Then V(J const ) = (1, 0) and V(J flip ) = (0, 1).Neither reads x in a way that depends on c.”

May combine two claims.

James et al. score overstatement as one continuous value, but their appendix decomposes linguistic certainty into extent, number, framing and probability, and reports that increased overstatement is driven by greater use of probability and extent certainty.

“James et al. [25] score overstatement as one continuous value, but their appendix decomposes linguistic certainty into extent, number, framing and probability, and reports that increased overstatement is driven by greater use of probability and extent certainty.”

• cites [25]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Expressed uncertainty has been located as a linear feature in representation space and steered along it.

“expressed uncertainty has since been located as a linear feature in representation space and steered along it [26]”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hedging is a studied feature of scientific writing with a documented rhetorical function.

“Hedging is a studied feature of scientific writing with a documented rhetorical function [23]”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

An item enters the sensitivity arm only at agreement ≥ 0.75 (pre-set).

“An item enters the sensitivity arm only at agreement ≥ 0.75 (pre-set; HUMAN_DIRECTION_AGREEMENT_MIN).”

May combine two claims.

If a language model adjudicates whether an intervention changes the correct verdict, then the evaluation tests models against models and any asymmetry could be a shared blind spot rather than a property of judges.

“If a language model adjudicates that, we are testing models against models and any asymmetry could be a shared blind spot rather than a property of judges.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Annotators see the pair without knowing which member is the edit, which axis it belongs to, or what any model said, and answer only whether the second sentence claims more than the evidence licenses.

“Annotators see the pair without knowing which member is the edit, which axis it belongs to, or what any model said, and answer only whether the second sentence claims more than the evidence licenses.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The per-axis form of the prediction is not testable on these sets and remains open.

“The per-axis form of the prediction is not testable on these sets and remains open.”

May combine two claims.

Reliability is not the binding constraint on judge quality; the two coordinates place no bound on each other.

“That reliability is the binding constraint on judge quality, assumed by the invariance literature and false by Proposition 1, which shows the two coordinates place no bound on each other.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The failure is structured, and the structure is the result.

“The failure is structured, and the structure is the result.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Shortcut exploitation is a mature literature on the model side, and the paper's control arm is a replication that exists to make the sensitivity arm interpretable.

“Shortcut exploitation is a mature literature on the model side ( §3), and our control arm is a replication that exists to make the sensitivity arm interpretable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Linguistic calibration has been trained for directly.

“Linguistic calibration (making an agent's expressed confidence match its competence) has been trained for directly [36]”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

What is new is measuring the second half (construct sensitivity) for evaluators and finding that it does not follow from the first (invariance).

“What is new is measuring the second half for evaluators and finding that it does not follow from the first.”

May combine two claims.

Test-retest consistency and human agreement are properties of J at fixed input; S extends them to perturbed input. By Proposition 1 none of the three constrains R, so the finding that judges are unreliable and the finding that they are insensitive are independent facts about the same instrument, and an intervention that raises agreement has no predicted effect on R at all.

“Test-retest consistency and human agreement are properties of J at fixed input; S extends them to perturbed input.By Proposition 1 none of the three constrains R.So the finding that judges are unreliable [38] and the finding that they are insensitive are independent facts about the same instrument, and an intervention that raises agreement (ensembling, self-consistency, rubric refinement [20]) has”

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

A calibration result about generation places no constraint on evaluator sensitivity.

“A calibration result about generation places no constraint on evaluator sensitivity.”

Classified as probably not a substantial claim. May combine two claims.

Both S and R are properties of the triple (judge, distribution, family); a profile is reported with its families named or it is not interpretable.

“Both are properties of the triple (judge, distribution, family): a profile is reported with its families named or it is not interpretable.”

May combine two claims.

The two axes respond with opposite sign to accuracy pressure, which supplies a mechanism for the reported paradox that prompting for accuracy increases overgeneralisation; pooling the axes hides it.

“The two axes further respond with opposite sign to accuracy pressure, which supplies a mechanism for the reported paradox that prompting for accuracy increases overgeneralisation [41]: pooling the axes hides it.”

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: For the constructs studied, the correct verdict y* is a fact about the relation between the claim and its evidence rather than about any annotator's opinion.
• AssumptionRests on an assumption: Human annotators can establish the direction of construct-changing interventions at agreement ≥ 0.75.
• AssumptionRests on an assumption: Drawing generation, verification, and judging from three disjoint model families prevents circularity.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ou et al. is adjacent in material rather than in question, asking how well grounded LLM critiques of papers are.

“Ou et al. [39] is adjacent in material rather than in question, asking how well grounded LLM critiques of papers are.”

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A second submission by an overlapping author set characterises the corpus that supplies the paper's probe items and C4's case study; it adopts this paper's decomposition and construct-blind predictor family, and reports corpus-side measurements; this paper reports judge-side ones. Neither depends on the other's results.

“A second submission by an overlapping author set characterises the corpus that supplies our probe items and C4's case study [4].It adopts this paper's decomposition and its construct-blind predictor family, and reports the corpus-side measurements; this paper reports the judge-side ones.Neither depends on the other's results, and the one quantity that would cross between them, a bound on judge”

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

No monotone scalar separates valid from degenerate; any symmetric scalar summary assigns the same score to (1,0) and (0,1), so a threshold admitting one degenerate judge admits the other.

“Corollary 1 (No monotone scalar separates valid from degenerate).Let g : [0, 1] 2 → R be any function non-decreasing in each argument, and suppose g assigns a passing score to some judge with V = (s * , r * ), s * , r * < 1.If g is symmetric (as a weighted mean with λ = 1 2 , or a harmonic mean, is) then g(1, 0) = g(0, 1), and any threshold that admits one degenerate judge admits the other.”

Probably not a substantial claim: may not pass S2 (checkable).

The construct of claim specificity is well posed.

“because the construct is well posed [27,41]”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: The constructs most exposed are those whose labels are expensive and whose surface correlates are cheap.

Classified as probably not a substantial claim.

PReMISE audits the rubric and the judge, not the label set the judge is scored against.

“Where it stops is where §4.4 starts: PReMISE audits the rubric and the judge, not the label set the judge is scored against, and Definition 6 is the quantity for that set.”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Every item in the T C arm is supported on both sides of the pair and differs only in what it covers or how hard it commits.

“every item in the T C arm is supported on both sides of the pair and differs only in what it covers or how hard it commits.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

An intervention belongs to the sensitivity arm only if it changes the correct verdict, and if a model decides that, then the experiment tests a model against itself.

“An intervention belongs to the sensitivity arm only if it changes the correct verdict, and if a model decides that, then the experiment tests a model against itself.”

Classified as probably not a substantial claim. May combine two claims.

Probing work on the strength of evidence supporting a clinical claim recovered a graded label linearly in every model tested and found the recoverable signal was largely lexical and did not transfer across topics.

“Probing work on a neighbouring construct, the strength of evidence supporting a clinical claim, recovered a graded label linearly in every model tested and then found the recoverable signal was largely lexical and did not transfer across topics [48].”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Park et al. differ from this paper: it is multimodal and about visual-versus-textual conflict; intended direction is established by construction rather than human adjudication; its object is mitigation via reward modelling rather than measurement, so it reports no invariance arm and therefore no profile.

“Three things separate it from §4: it is multimodal and about visual-versus-textual conflict; the intended direction is established by construction rather than by human adjudication, which is the circularity Protocol 1 closes; and its object is mitigation via reward modelling rather than measurement, so it reports no invariance arm and therefore no profile.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Direction is set by human annotators at agreement ≥ 0.75, and generation, verification, and judging are three disjoint model families.

“Direction is therefore set by human annotators, at agreement ≥ 0.75, and generation, verification, and judging are three disjoint model families.”

• AssumptionRests on an assumption: Human annotators can establish the direction of construct-changing interventions at agreement ≥ 0.75.
• AssumptionRests on an assumption: Drawing generation, verification, and judging from three disjoint model families prevents circularity.

May combine two claims.

The unification in Jiang et al. is what buys the conformal guarantee; for generation under a coverage guarantee it is the right modelling choice.

“The unification is what buys the conformal guarantee, since a single nested family of answer sets is what a calibrated set predictor needs, and for generation under a coverage guarantee it is the right modelling choice.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Decontextualising a scientific snippet is a defined task with gold data.

“decontextualising a scientific snippet (lifting it off the source without widening what it covers) is a defined task with gold data [37]”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Zhu reports that models inconsistently hedge and decline generalisations reflecting documented patterns; generic formation is the scope operation and hedging the strength one, and a model turning both inconsistently is not what a single unstructured dial produces.

“Studying whether a chatbot should assert generics about social groups, Zhu [61] reports that models inconsistently hedge and decline generalisations reflecting documented patterns.Generic formation is the scope operation and hedging the strength one, and a model turning both inconsistently is not what a single unstructured dial produces.”

• cites [61]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The paper measures the profile over 7 judges and 4 domains, against minimal interventions along 7 construct slots and 5 register-only controls, with direction established by human annotators at a pre-registered agreement bar of 0.75, and generation, verification and judging drawn from three disjoint model families.

“We then measure it, over 7 judges and 4 domains, against minimal interventions along 7 construct slots and 5 register-only controls, with direction established by human annotators rather than by any model at a pre-registered agreement bar of 0.75, and generation, verification and judging drawn from three disjoint model families.”

• AssumptionRests on an assumption: Human annotators can establish the direction of construct-changing interventions at agreement ≥ 0.75.
• AssumptionRests on an assumption: Drawing generation, verification, and judging from three disjoint model families prevents circularity.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Park et al. build minimally edited counterfactual responses isolating perceptual errors, catching a judge that fails to downgrade them.

“Park et al. [40] build minimally edited counterfactual responses isolating perceptual errors, catching a judge that fails to downgrade them.”

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The probability and extent axes in James et al. are the paper's scope and strength under other names, found in their data and discarded by the headline metric.

“Those are our two axes under other names, found in their data and discarded by the headline metric.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

A judge insensitive on strength may be perfectly serviceable where strength is not the construct, and human labels are not ground truth in general.

“A judge insensitive on strength may be perfectly serviceable where strength is not the construct, and human labels are not ground truth in general [24].”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

Overclaiming is not one thing; the works closest to ours reach the scope/strength distinction and then unify it deliberately, theoretically and empirically.

“That overclaiming is one thing, assumed by the works closest to ours, which reach the scope/strength distinction and then unify it deliberately, theoretically [27] and empirically [25]; we treat the unification as a hypothesis with a refutation condition.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [25]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The two axes have been noticed before, twice, and both times deliberately collapsed; the paper claims that the collapse costs something measurable.

“The two axes have been noticed before, twice, and both times deliberately collapsed: we claim that the collapse costs something measurable, which is narrower and more checkable than novelty ( §3.1).”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [25]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Roy et al. treat judge rubrics as measurement specifications and audit them for structural adequacy, reliability, preference fit and adversarial robustness, concluding that no source is simultaneously reliable, preference-predictive and robust, and that high inter-rater agreement does not prevent exploitability.

“Roy et al. [46] treat judge rubrics as measurement specifications and audit them for structural adequacy, reliability, preference fit and adversarial robustness, concluding that no source is simultaneously reliable, preference-predictive and robust, and that high inter-rater agreement does not prevent exploitability.”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Every annotation unit is one sentence with its evidence context, presented with no indication of which side of a pair it came from, no indication of the producing system, sentences from both sides interleaved and globally shuffled so pair membership is unrecoverable, and the length cue broken by presenting sentences individually rather than whole responses.

“Every annotation unit is one sentence with its evidence context, presented with no indication of which side of a pair it came from, no indication of the producing system, sentences from both sides interleaved and globally shuffled so pair membership is unrecoverable, and the length cue broken by presenting sentences individually rather than whole responses.”

• Also stated in: § 5.5 What it costs: a blinded replacement, and ∆ · plain

  “Right: the unit is one sentence with its evidence context, every field that identifies its side withheld, sides interleaved so pair membership is unrecoverable, and the label the majority of three independent annotators with no arbitration pass.”

Classified as probably not a substantial claim.

A slot or axis below the bar is reported as not measurable and excluded from R rather than reported with a wider interval, because a wide interval around an ill-defined construct still asserts the construct exists.

“a slot or axis below the bar is reported as not measurable and excluded from R rather than reported with a wider interval, because a wide interval around an ill-defined construct still asserts the construct exists.”

Probably not a substantial claim: may not pass S4 (based).

Definition 5 predicts the reverse ordering, since labels further from the generative condition should be harder to reach without the construct.

“Definition 5 predicts the reverse ordering, since labels further from the generative condition should be harder to reach without the construct.”

Probably not a substantial claim: may not pass S2 (checkable).

§1's apparent contradiction is exactly a well-behaved judge meeting a leaky ruler.

“§1's apparent contradiction is exactly a well-behaved judge meeting a leaky ruler.”

Classified as probably not a substantial claim. May combine two claims.

That is a generator yield problem rather than a construct problem, so the slot stays in scope with its yield stated.

“That is a generator yield problem rather than a construct problem, so the slot stays in scope with its yield stated.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

If strength edits are harder to construct cleanly than scope edits, then the surviving strength items passed a stricter filter and part of any asymmetry is due to the pipeline rather than the judges.

“if strength edits are harder to construct cleanly than scope ones then the surviving strength items passed a stricter filter and part of any asymmetry is the pipeline rather than the judges.”

Classified as probably not a substantial claim.

Three further checks, all specified before the numbers existed, hold.

“Three further checks, all specified before the numbers existed, hold.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every measured number below is generated from results/ rather than typed into the manuscript.

“Every measured number below is generated from results/ rather than typed into the manuscript, and the two arms that remain unrun are named as such where they would appear.”

Classified as probably not a substantial claim. May combine two claims.

Two separate problems were stacked there, and only the annotators could have separated them.

“Two separate problems were stacked there, and only the annotators could have separated them.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A set whose label is a function of a relative property is the reverse.

“A set whose label is a function of a relative property is the reverse.”

May combine two claims.

RewardBench pairs are built so the two sides differ within a pair, but across the set no absolute length band marks the chosen side, so a per-item predictor is near chance and a paired one is not.

“RewardBench pairs are built so the two sides differ within a pair, but across the set no absolute length band marks the chosen side, so a per-item predictor is near chance and a paired one is not.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

A provenance-inherited label set for claim specificity already exists and is known to leak.

“a provenance-inherited label set for it already exists and is known to leak [5]”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The design floor is six judge families spanning vendors, parameter scales, and reasoning-enabled versus not, over at least four evaluation domains.

“The design floor is six judge families spanning vendors, parameter scales, and reasoning-enabled versus not, over at least four evaluation domains.”

May combine two claims.

If ensembling and self-consistency raise S without moving R, then the field's standard reliability remedies do not touch validity, and Proposition 1 has an empirical instance rather than only a proof.

“If ensembling and self-consistency raise S without moving R, then the field's standard reliability remedies do not touch validity, and Proposition 1 has an empirical instance rather than only a proof.”

Classified as probably not a substantial claim. May combine two claims.

Restricting the study to one construct in one domain would make the finding a fact about geoscience claim-scope rather than about evaluators, so the profile is measured across domains whose criteria differ in kind, with the scientific-claim domain as the one where minimal edits are best posed and human verification is cheapest.

“Restricting the study to one construct in one domain would make the finding a fact about geoscience claim-scope rather than about evaluators, so the profile is measured across domains whose criteria differ in kind, with the scientific-claim domain as the one where minimal edits are best posed and human verification is cheapest.”

Classified as probably not a substantial claim.

VP is undefined until the mode is fixed.

“VP is undefined until the mode is fixed.”

May combine two claims.

Because the scope edit is the longer member, the length bias pushes judges toward naming the original on scope items and the edit on strength ones.

“Since the scope edit is the longer member, that bias pushes judges toward naming the original on scope items and the edit on strength ones.”

May combine two claims.

Reporting the lower of the two modes is not conservative; it is simply the answer to whichever question was not asked.

“Hence a single reported a imp without its mode is uninterpretable, and reporting the lower of the two is not conservative; it is simply the answer to whichever question was not asked.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

R is uninterpretable without S.

“It is here because R is uninterpretable without S (Proposition 2), which makes the replication a measurement requirement rather than a contribution.”

May combine two claims.

The confound was not removed; its sign was flipped and its magnitude preserved, and a reward model that learned to prefer brevity scores above chance on the benchmark built to detect it.

“The confound was not removed; its sign was flipped and its magnitude preserved, and a reward model that learned to prefer brevity scores above chance on the benchmark built to detect it.”

Classified as probably not a substantial claim. May combine two claims.

3 annotators independently labelled all 478 items; the third worked the full batch rather than arbitrating the first two's disagreements, so the three label sets are independent.

“3 annotators independently labelled all 478 items; the third worked the full batch rather than arbitrating the first two's disagreements, so the three label sets are independent.”

May combine two claims.

Within-set recovery is necessary but not sufficient.

“Within-set recovery is necessary but not sufficient [48]”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The released harness enforces three requirements: agreement is chance-corrected, each a_imp is reported against a label-permutation null, and a_hum carries its own interval.

“Three requirements, all of which the released harness enforces.Agreement is chance-corrected, so VP is not inflated by class imbalance.Each a imp is reported against a label-permutation null, so a small positive VP is distinguishable from noise.And a hum carries its own interval, because a ceiling estimated from few double-annotated items can be the larger source of uncertainty in Definition 6.”

May combine two claims.

The public benchmarks are no longer the clean comparison they appeared to be in the per-item sweep: they sit well above chance, not at it.

“What has changed is that the public benchmarks are no longer the clean comparison they appeared to be in the per-item sweep: they sit well above chance, not at it.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

If C is recoverable and y = g(C), then y is reachable without the construct, and this predictor bounds how much of L any surface-reading instrument can obtain.

“If C is recoverable and y = g(C), then y is reachable without the construct, and this predictor bounds how much of L any surface-reading instrument can obtain.”

Classified as probably not a substantial claim. May combine two claims.

The pool and the pre-set bar of 0.75 are specified in Appendix C, frozen before any data was collected.

“The pool and the pre-set bar of 0.75 are specified in Appendix C, frozen before any data was collected”

Classified as probably not a substantial claim. May combine two claims.

The gap was to be reported as refuted if it came in below 0.10 or failed a paired bootstrap over base items.

“The sign, the magnitude and the condition that would have refuted them were all registered before measurement (Appendix F): the gap was to be reported as refuted if it came in below 0.10 or failed a paired bootstrap over base items.”