Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

May combine two claims.

A pre-registered replication on 300 fresh questions confirms the dissociation and the rule's effect.

“A pre-registered replication on 300 fresh questions confirms the dissociation and the rule's effect.”

• Also stated in: § 6. The pattern replicates on fresh questions, a second task and harder failures · p. 8 · boosted

  “All nineteen tests pass (Figure 3, in parentheses; Appendix D.4).”

May combine two claims.

After judging five consecutive results useless, unaided Qwen3-8B answers on only 1 of 292 questions, and the other models in Figure 1 answer on at most 7%.

“After judging five consecutive results useless, unaided Qwen38B answers on only 1 of 292 questions, and the other models in Figure 1 answer on at most 7%.”

May combine two claims.

Stopping at the point after five consecutive useless judgments would have paid, since answering there succeeds 15-17% of the time and continuing at most 7% for 7-8B models.

“Stopping there would have paid, since answering at that point succeeds 15-17% of the time and continuing at most 7% (7-8B models, exploratory; Appendix D.2).”

May combine two claims.

When Δ is computed from stated judgments, it stays below zero, and after five stated useless judgments in a row the agents answer on 0-0.4% of questions (Qwen3-32B: 6%).

“When Δ is computed from these stated judgments instead (Appendix D.4), it stays below zero, and after five stated useless judgments in a row the agents answer on 0-0.4% of questions (Qwen3-32B: 6%).”

May combine two claims.

On a failing source, Qwen3-8B overestimates the value of continuing, Qwen2.5-7B overestimates that of answering now, and Llama-3.1-8B underestimates it.

“On a failing source, Qwen3-8B does overestimate the value of continuing, but Qwen2.5-7B overestimates that of answering now and Llama-3.1-8B underestimates it (descriptive probe, Appendix C.4).”

May combine two claims.

Qwen3-32B's integration index rises to +0.32, so the pre-registered prediction that it would stay near zero failed.

“Its integration index rises to + 0 . 32, so our pre-registered prediction that it would stay near zero failed.”

May combine two claims.

On fresh questions, Δ stays below zero for the unaided and budget-cued agents and reaches +0.30 to +0.35 under the enforced rule, which beats the unaided agent on mean6 (+0.030 to +0.061).

“Again, Δ stays below zero for the unaided and budget-cued agents and reaches + 0 . 30 to + 0 . 35 under the enforced rule, which beats the unaided agent on mean6 ( + 0 . 030 to + 0 . 061).”

May combine two claims.

Against the best fixed cap or run-length policy chosen in hindsight, unaided agents lose 6-13 points of success, and with a doubled budget the budget-cued 7-8B Qwen models make 61-71% more calls for no gain.

“Against the best fixed cap or run-length policy chosen in hindsight, unaided agents lose 6-13 points of success (exploratory; Appendix D.2), and with a doubled budget the budget-cued 7-8B Qwen models make 61-71% more calls for no gain.”

May combine two claims.

Against a random signal that says 'useless' equally often but at random steps, the enforced rule wins by 0.011 to 0.029 (Holm-significant), so the timing of the signal matters.

“Against a random signal that says 'useless' equally often but at random steps, the enforced rule wins by 0 . 011 to 0 . 029 (Holm-significant), so the timing of the signal matters.”

Classified as probably not a substantial claim. May combine two claims.

Search-R1: judgment validity, USELESS after failed ≥ .80 and after real ≤ .50, else Δ only descriptive. Outcomes: fail: 1.00 and .89.

“fail : 1.00 and .89”

Classified as probably not a substantial claim. May combine two claims.

fresh300: own-judgment Δ UB < +.06 for none and budget; 7-8B (Qwen3-32B if time allows). Outcomes: pass 8/8: none -.09/-.03/-.05, Qwen3-32B -.06; budget -.17/-.14/-.19, Qwen3-32B -.05 (UB +.01; others <0).

“pass 8/8: none - . 09 / - . 03 / - . 05, Qwen3-32B - . 06; budget - . 17 / - . 14 / - . 19, Qwen3-32B - . 05 (UB + . 01; others < 0)”

May combine two claims.

With the price stated, Claude Haiku 4.5 answers on almost every question regardless of the evidence and even rivals the enforced rule's success.

“With the price stated, it answers on almost every question regardless of the evidence and even rivals the enforced rule's success (Appendix D.3).”

May combine two claims.

Only Qwen2.5-7B gains appreciably on mean6 in the decide condition and even beats the enforced rule, but it does so by stopping early indiscriminately.

“Only Qwen2.5-7B gains appreciably on mean6 and even beats the enforced rule, but it does so by stopping early indiscriminately.”

May combine two claims.

When failed observations are plausible or answerless, judgments are less accurate (84-97% and 76-97% useless), but the dissociation remains.

“When the failed observations are the question's own distractor paragraphs ( plausible ) or its own pages with every supporting fact removed ( answerless ), judgments are less accurate (84-97% and 76-97% useless), but the dissociation remains.”

May combine two claims.

When the primary search fails throughout, Qwen3-8B and Qwen3-32B switch at the first useless result.

“When the primary search fails throughout, Qwen3-8B and Qwen3-32B reasonably switch at the first useless result.”

May combine two claims. The quote only partly states this claim.

fresh300: unaided agent answers on <10% of questions after five own useless judgments (persistent). Outcomes: pass 3/3: 3/125, 2/283, 0/289.

“pass 3/3: 3/125, 2/283, 0/289”

Classified as probably not a substantial claim.

The pattern replicates.

“The pattern replicates (§6)”

May combine two claims.

Because Search-R1's judgments call most real results useless (89%), the paper does not interpret its Δ; the enforced rule still raises its mean6 by +0.024.

“Because its judgments also call most real results useless (89%), we do not interpret its Δ ; the enforced rule still raises its mean6 by + 0 . 024 (Appendix D.4).”

The quote only partly states this claim.

The pattern holds for larger, reasoning and search-trained agents.

“Larger, reasoning and search-trained agents (§7)”

Classified as probably not a substantial claim. May combine two claims.

Both exits named: backup share of first exits > .6 / < .4; 7-8B, Qwen3-32B thinking mode, four regimes: own-judgment Δ CI. Outcomes: prefers evidence: Qwen2.5-7B .93, Llama-3.1-8B .93, Qwen3-32B .90; no clear preference: Qwen3-8B .52. pass 2/2: none +.01 [-.18,+.17]; budget +.25 [-.01,...].

“prefers evidence: Qwen2.5-7B .93, Llama-3.1-8B .93, Qwen3-32B .90; no clear preference: Qwen3-8B .52 pass 2/2: none + . 01 [ - . 18, + . 17]; budget + . 25 [ - . 01,”

May combine two claims.

By the fourth action, Llama-3.1-8B and Qwen2.5-7B have switched on only 17% and 49% of questions, and Llama-3.1-8B's success falls to 0.05 (0.50 with a working primary).

“By the fourth action, however, Llama-3.1-8B and Qwen2.57B have switched on only 17% and 49% of questions, and Llama-3.1-8B's success falls to 0.05 (0.50 with a working primary).”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The enforced rule does not lose success where the source recovers.

“without loss where the source recovers”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The enforced rule has a stopping point that does not follow the budget.

“with a stopping point that does not follow the budget”

May combine two claims.

On a failing source, Search-R1 answers from memory after one or two searches on 27% of questions but searches to the end of its budget on 45%.

“On a failing source it answers from memory after one or two searches on 27% of questions but searches to the end of its budget on 45%.”

May combine two claims.

In the decide condition, agents still do not stop on the count, and their Δ is -0.14 to -0.24 for the three 7-8B models.

“Agents still do not stop on the count, and their Δ is -0 . 14 to -0 . 24 for the three 7-8B models (exploratory; Figure 3).”

May combine two claims.

With its budget stated, Qwen3-32B answers around the sixth action and matches the enforced rule's success.

“With its budget stated, Qwen3-32B answers around the sixth action and matches the enforced rule's success.”

May combine two claims.

With the budget doubled, budget-cued agents move their answers from the eighth to the sixteenth action and roughly double their tool calls on a failing source without any gain in success.

“With the budget doubled, budget-cued agents move their answers from the eighth to the sixteenth action and roughly double their tool calls on a failing source without any gain in success (Figure 4).”

May combine two claims.

In same-trajectory dissociation (unaided; own replayed judgments), unaided agents answer on few questions after five own useless judgments: Qwen2.5-7B 3/122, Llama-3.1-8B 1/282, Qwen3-8B 1/292, Qwen3-32B 19/284.

“reported: answers on 3/122, 1/282, 1/292; 19/284”

May combine two claims.

With the rule stated in words, only Llama-3.1-8B and Qwen3-32B follow it, and only partly (Δ = +0.09 and +0.08, against +0.37 and +0.33 when enforced).

“With the rule stated in words, only Llama-3.1-8B and Qwen3-32B follow it, and only partly ( Δ = + 0 . 09 and + 0 . 08, against + 0 . 37 and + 0 . 33 when enforced).”

May combine two claims.

Own-judgment Δ CI includes or lies below 0 for none, permit, budget, policy, cost; LB > 0 for rule.

“Own-judgment Δ : CI includes or lies below 0 for none, permit, budget, policy, cost; LB > 0 for rule”

May combine two claims.

Stating the budget with a counter makes the 7-8B models answer on the last action (51-79% of answers on a failing source), and their Δ stays negative.

“Stating the budget with a counter does make the 7-8B models answer, but on the last action (51-79% of answers on a failing source), and their Δ stays negative.”

May combine two claims.

Two sources: switched by action 4 ≥ .5 ⇒ 'switches'; else 'does not abandon' if own-judgment Δ for leaving includes or lies below 0; 7-8B, Qwen3-32B. Outcomes: Qwen2.5-7B, Llama-3.1-8B: does not abandon (.49, .17; Δ -.54, -.06); Qwen3-8B, Qwen3-32B: switches (.73, .95).

“Qwen2.5-7B, Llama-3.1-8B: does not abandon (.49, .17; Δ - . 54, - . 06); Qwen3-8B, Qwen3-32B: switches (.73, .95)”

May combine two claims.

Where the source recovers or never fails, the enforced rule is non-inferior to the unaided agent in 18 of 20 tests (margin -0.05; Qwen3-8B fails narrowly twice).

“Where the source recovers or never fails, the rule is non-inferior to the unaided agent in 18 of 20 tests (margin -0 . 05; Qwen3-8B fails narrowly twice).”

May combine two claims.

The enforced rule raises Claude Sonnet 5's success on a failing source from 0.40 to 0.51, although on 100 questions the gain over six regimes is not significant (+0.022, p = 0.051).

“The enforced rule raises its success on a failing source from 0.40 to 0.51, although on 100 questions the gain over six regimes is not significant ( + 0 . 022, 𝑝 = 0 . 051).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Stated cost: Δ CI includes or lies below 0; combo no longer called cost-optimal if cost arm's cost-adjusted success LB > -0.02 vs. combo.

“Stated cost: Δ CI includes or lies below 0; combo no longer called cost-optimal if cost arm's cost-adjusted success LB > - 0 . 02 vs. combo”

Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

fresh300: rule > none (mean6), one-sided, Holm. Outcomes: pass 4/4: +.030 / +.061 / +.036; Qwen3-32B +.057 (all p < .001).

“pass 4/4: + . 030 / + . 061 / + . 036; Qwen3-32B + . 057 (all 𝑝 < . 001)”

May combine two claims.

Agents recognize useless results, but their beliefs, what they are told and the running count all fail to make the 7-8B models stop on them, apart from Llama-3.1-8B's partial following of the stated rule.

“Agents recognize useless results, but their beliefs, what they are told and the running count all fail to make the 7-8B models stop on them, apart from Llama-3.1-8B's partial following of the stated rule.”

May combine two claims.

On FEVER, unaided agents answer after five useless judgments on at most 3% of claims, and the enforced rule raises mean6 by +0.129 to +0.201.

“Unaided agents answer after five useless judgments on at most 3% of claims, and the enforced rule raises mean6 by + 0 . 129 to + 0 . 201.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

fresh300: rule Δ LB > +.20. Outcomes: pass 4/4: +.31 [LB +.22] / +.35 [+.33] / +.34 [+.31]; Qwen3-32B +.30 [+.26].

“pass 4/4: + . 31 [LB + . 22] / + . 35 [ + . 33] / + . 34 [ + . 31]; Qwen3-32B + . 30 [ + . 26]”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The stopping point can be read from the agent's reasoning without a judgment call.

“with no judgment call, read from the agent's reasoning”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Long recovery: rule < none in rec5; rule ≥ none -0.05 in rec4. Outcomes: rec5 holds for Qwen3-8B (-.213), Qwen3-32B (-.137), fails for Qwen2.5-7B, Llama-3.1-8B; rec4: 3/4, fail Qwen3-8B (LB -.050).

“rec5: holds for Qwen3-8B ( - . 213), Qwen3-32B ( - . 137), fails for Qwen2.5-7B, Llama-3.1-8B; rec4: 3/4, fail Qwen3-8B (LB - . 050)”

May combine two claims.

The enforced rule still raises success on plausible and answerless sources, although on the answerless source the budget condition does better.

“The enforced rule still raises success in both, although on the answerless source the budget condition does better (Appendices C.1, D.5).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Third run: Rp1-Rp4: Rp1 3/3; Rp2 fail Qwen2.5-7B (as before); Rp3 3/3; Rp4 6/6.

“Rp1 3/3; Rp2 fail Qwen2.5-7B (as before); Rp3 3/3; Rp4 6/6”

May combine two claims.

Qwen3-32B judges failing results as accurately as the smaller models, yet answers after five such judgments on only 19 of 284 questions.

“Qwen3-32B judges failing results as accurately as the smaller models, yet answers after five such judgments on only 19 of 284 questions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Deadline and integration combine.

“Deadline and integration combine (§5)”

May combine two claims.

Switch rule - none, mean over six two-source regimes: Llama-3.1-8B +.054, Qwen3-8B +.016 (both LB >0); Qwen2.5-7B +.011, Qwen3-32B +.007 (n.s.).

“Llama-3.1-8B + . 054, Qwen3-8B + . 016 (both LB > 0); Qwen2.5-7B + . 011, Qwen3-32B + . 007 (n.s.)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The enforced rule makes three to five judgment calls per question, which would put it below the budget condition if they were charged like tool calls.

“The enforced rule also makes three to five judgment calls per question, which would put it below the budget condition if they were charged like tool calls.”

May combine two claims.

At the same step, Qwen3-32B answers as often after earlier evidence it judged useful as after a run it judged useless (Δ = 0.00 [-0.06, +0.05]), and with the budget doubled it answers a step later and makes 46% more tool calls for no gain.

“At the same step, Qwen3-32B answers as often after earlier evidence it judged useful as after a run it judged useless ( Δ = 0 . 00 [ -0 . 06 , + 0 . 05]), and with the budget doubled it answers a step later and makes 46% more tool calls for no gain.”

May combine two claims.

The combination of budget and enforced rule is non-inferior to both components on every model and task (14 of 14 pre-registered tests), and it beats the enforced rule by +0.138 for Qwen2.5-7B.

“The combination is indeed non-inferior to both components on every model and task (14 of 14 pre-registered tests), and it beats the enforced rule by + 0 . 138 for Qwen2.5-7B, whose main failure is answering too early.”

Classified as probably not a substantial claim. May combine two claims.

Stated vs. side-channel judgments: annotator binary κ ≥ .80 on adjudicated labels to proceed; agreement ≥ .85 expected. Outcomes: κ .874 [.798,.939]; agreement .87/.94/.94; Qwen3-32B .96.

“𝜅 .874 [.798, .939]; agreement .87 / .94 / .94; Qwen3-32B .96”

Classified as probably not a substantial claim. May combine two claims.

k=5 maximized development mean6 for Qwen3-8B and Llama-3.1-8B (0.405, 0.268), and Qwen2.5-7B's development sweep also peaked at k=5 (0.192).

“𝑘 = 5 maximized development mean6 for Qwen3-8B and Llama-3.1-8B (0.405, 0.268), and Qwen2.5-7B's development sweep also peaked at 𝑘 = 5 (0.192).”

May combine two claims.

Permission makes Qwen3-8B stop earlier, but its stops ignore the evidence; it answers before any real result on 55% of questions whose source recovers after three failures, and its Δ drops to -0.32.

“Permission does make Qwen3-8B stop earlier, but its stops ignore the evidence. The model answers before any real result on 55% of questions whose source recovers after three failures, and its Δ drops to -0 . 32.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The step is cheap, because the harness can read the judgment in the agent's reasoning instead of asking for it.

“The step is also cheap, because the harness can read the judgment in the agent's reasoning instead of asking for it.”

May combine two claims.

No unaided agent becomes more likely to leave as useless results accumulate (Δ ≤ +0.02), whereas disabling the failing tool after five useless judgments makes leaving evidencedriven (Δ = +0.26 and +0.30 for Llama-3.1-8B and Qwen3-8B).

“Yet no unaided agent becomes more likely to leave as useless results accumulate ( Δ ≤ + 0 . 02), whereas disabling the failing tool after five useless judgments makes leaving evidencedriven ( Δ = + 0 . 26 and + 0 . 30 for Llama-3.1-8B and Qwen3-8B; Appendix D.5).”

May combine two claims.

With a price of 0.05 per call added to the budget, the 7-8B agents answer at the deadline and do no better than with the budget alone; only Qwen3-32B stops partly on its judgments.

“With a price of 0.05 per call added to the budget, the 7-8B agents simply did not know which stopping rule models still answer at the deadline and do no better than with the budget alone; only Qwen3-32B stops partly on its judgments.”

May combine two claims.

Search-R1: rule > none (mean6), one-sided. Outcomes: pass: +.024 [+.009,+.041].

“pass: + . 024 [ + . 009, + . 041]”

Probably not a substantial claim: may not pass S3 (load-bearing).

The enforced rule steps up at j = 5.

“The enforced rule (blue) steps up at 𝑗 = 5”

May combine two claims.

Δ from stated judgments (unaided, test300): UB < +.06. Outcomes: pass 4/4: -.08/-.01/-.04; Qwen3-32B -.02 (UB +.002).

“pass 4/4: - . 08 / - . 01 / - . 04; Qwen3-32B - . 02 (UB + . 002)”

May combine two claims.

The unaided agent answered on 69 of 100 persistent questions, with I +0.18 [+0.11,+0.26] (rule: +0.49); on the 65 questions where the rule fired, the unaided agent exhausted its budget on 29 (45%; Claude Haiku 4.5: 282 of 284).

“The unaided agent answered on 69 of 100 persistent questions, with 𝐼 + 0 . 18 [ + 0 . 11, + 0 . 26] (rule: + 0 . 49); on the 65 questions where the rule fired, the unaided agent exhausted its budget on 29 (45%; Claude Haiku 4.5: 282 of 284).”

May combine two claims.

Agents do not always write a judgment: the reader finds none at 0.25, 0.15 and 0.22 of decision points for Llama-3.1-8B, Qwen3-8B and Qwen3-32B, but at 0.74 for Qwen2.5-7B, whose thoughts are mostly empty (0.39 when told its budget).

“Agents do not always write a judgment: the reader finds none at 0.25, 0.15 and 0.22 of decision points for Llama-3.1-8B, Qwen3-8B and Qwen3-32B, but at 0.74 for Qwen2.5-7B, whose thoughts are mostly empty (0.39 when told its budget).”

May combine two claims.

Stated-judgment rule ≥ enforced rule -0.05 (mean6, 95% LB). Outcomes: pass 3/3 applicable (lowest LB -.017); Qwen2.5-7B LB -.041.

“pass 3/3 applicable (lowest LB - . 017); Qwen2.5-7B LB - . 041”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B complied least (rule arm: 42 of 105).

“Qwen2.5-7B complied least (rule arm: 42 of 105).”

May combine two claims.

Accuracy of stated judgments: Qwen3-8B states in its own reasoning call useless evidence useless 93.9% of the time (n=1,463) and treat useful or partly useful evidence as such 98.1% of the time (n=212).

“the judgments Qwen3-8B states in its own reasoning call useless evidence useless 93.9% of the time ( 𝑛 = 1 , 463) and treat useful or partly useful evidence as such 98.1% of the time ( 𝑛 = 212).”

May combine two claims.

Thinking minus non-thinking, mean over the four regimes: none +0.076 [+0.038, +0.112], budget -0.051 [-0.083, -0.018].

“Thinking minus non-thinking, mean over the four regimes: none + 0 . 076 [ + 0 . 038 , + 0 . 112], budget -0 . 051 [ -0 . 083 , -0 . 018] (Table 24).”

Classified as probably not a substantial claim. May combine two claims.

Pre-registered predictions: Δ's interval includes 0 or lies below it for the unaided and prompt arms, and lies above 0 for the rule. Both hold for the rule and for the unaided, permission and budget arms on all four models; they fail for the stated rule on Llama-3.1-8B and Qwen3-32B and for the stated cost on Qwen3-32B, which the main text reports as partly evidence-driven.

“Both hold for the rule and for the unaided, permission and budget arms on all four models; they fail for the stated rule on Llama-3.1-8B and Qwen3-32B and for the stated cost on Qwen3-32B, which the main text reports as partly evidence-driven.”

Classified as probably not a substantial claim.

Stated versus side-channel judgments: a local Qwen3-32B annotator labeled the 327 reference stated judgments at five-class κ 0.842 and binary 0.874 before being applied to four models' test trajectories.

“a local Qwen3-32B annotator (nonthinking, temperature 0, the same rubric) labeled the 327 reference stated judgments at five-class 𝜅 0.842 and binary 0.874 before being applied to four models' test trajectories”

May combine two claims. The quote only partly states this claim.

Utility at λ=0.05, all calls counted: stated-judgment rule > enforced rule; one-sided, Holm. Outcomes: pass 3/3 applicable: +.236/+.196/+.190; Qwen2.5-7B +.086.

“pass 3/3 applicable: + . 236 / + . 196 / + . 190; Qwen2.5-7B + . 086”

May combine two claims.

Qwen3-32B's design-label contrast is Δ = 0.00 (Figure 3); thinking mode's design-label contrast is -0.02 (Appendix C.1).

“Qwen3-32B: Δ = 0 . 00, Figure 3; thinking: design-label contrast -0 . 02, Appendix C.1”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claude Haiku 4.5 had 13 placeholders, Qwen3-32B 24, Claude Sonnet 5 none.

“Claude Haiku 4.5 had 13, Qwen3-32B 24, Claude Sonnet 5 none”

Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

Utility at λ=0.05: stated-judgment combination > budget; one-sided, Holm. Outcomes: 2/3 applicable: +.045/+.022; fail Qwen3-32B (.000); Qwen2.5-7B +.027.

“2/3 applicable: + . 045 / + . 022; fail Qwen3-32B ( . 000); Qwen2.5-7B + . 027”

Classified as probably not a substantial claim. May combine two claims.

Claude Haiku 4.5, stated rule vs. none and vs. enforced rule (mean6), reported. Outcomes: .043 / -.050.

“. 043 / - . 050”

May combine two claims.

Δ on the judgments recorded during the decide runs: Qwen2.5-7B -0.24 [-0.37,-0.13]; Llama-3.1-8B -0.14 [-0.18,-0.11]; Qwen3-8B -0.15 [-0.29,-0.01]. Decide was not run for Qwen3-32B.

“Δ on the judgments recorded during the decide runs (Figure 3): Qwen2.5-7B -0 . 24 [ -0 . 37 , -0 . 13]; Llama-3.1-8B -0 . 14 [ -0 . 18 , -0 . 11]; Qwen3-8B -0 . 15 [ -0 . 29 , -0 . 01]. Decide was not run for Qwen3-32B.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the second run, Rp4 for Qwen3-8B passes only after repair.

“in the second run, Rp4 for Qwen3-8B passes only after repair”

May combine two claims.

At the -0.02 margin drafted before amendment 6, the combination's non-inferiority to its components holds in 12 of 14 original tests; it fails for Qwen3-8B against the rule on HotpotQA (lower bound -0.023) and Qwen3-32B against the rule on HotpotQA (lower bound -0.022).

“At the -0 . 02 margin drafted before amendment 6, the combination's noninferiority to its components holds in 12 of 14 original tests; it fails for Qwen3-8B against the rule on HotpotQA (lower bound -0 . 023) and Qwen3-32B against the rule on HotpotQA (lower bound -0 . 022).”

May combine two claims.

Claude Haiku 4.5, rule stated in prompt (200 questions): executed/partial/not executed, by rule of PP1. Outcomes: partial: design-label contrast +.052 [+.021,+.085], median action 7, vs. enforced rule -.050 [LB -.086] +.

“partial : design-label contrast + . 052 [ + . 021 , + . 085], median action 7, vs. enforced rule - . 050 [LB - . 086] +”

May combine two claims.

Search-R1: own-judgment Δ UB < +.06 (none). Outcomes: formally pass (-.26 [-.58,-.06]); not interpreted (V).

“formally pass ( - . 26 [ - . 58, - . 06]); not interpreted (V)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Realized answer-now success on persistent: .12/.15/.14 raw, .14/.17/.19 repaired.

“Realized answer-now success on persistent: . 12/ . 15/ . 14 raw, . 14/ . 17/ . 19 repaired.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The first author adjudicated a random 40 of these and found the primary label correct on 35 (0.875, Wilson 95% [0.74,0.95]); no count changes.

“The first author also adjudicated a random 40 of these and found the primary label correct on 35 (0.875, Wilson 95% [0.74, 0.95]); no count changes.”

Classified as probably not a substantial claim. May combine two claims.

Reference set: two further LLM annotators independently labeled a stratified sample of 360 observations and 327 stated judgments; they agree at κ 0.875 and 0.868 (binary 0.972 and 0.898), and the first author adjudicated their disagreements.

“They agree at 𝜅 0.875 and 0.868 (binary 0.972 and 0.898), and the first author adjudicated their disagreements.”

May combine two claims.

The reasoning mode (Qwen3-8B) and the budget-cued Qwen3-32B rise with j, but not with the evidence at a fixed step.

“the reasoning mode (brown, Qwen3-8B) and the budget-cued Qwen332B rise with 𝑗 , but not with the evidence at a fixed step”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On HotpotQA test300 (2,100 trajectories per arm), placeholders concentrated in Qwen3-8B's stated-rule, budget, combination, rule, lexical and permit arms (297, 231, 226, 188, 146, 114), Llama-3.1-8B's stated-rule arm (149) and Qwen2.5-7B's rule and random arms (105, 82); the unaided arms had one in total, and the unrepaired call-cost arm 364 for Qwen3-8B.

“On HotpotQA test300 (2,100 trajectories per arm), placeholders concentrated in Qwen38B's stated-rule, budget, combination, rule, lexical and permit arms (297, 231, 226, 188, 146, 114), Llama-3.1-8B's stated-rule arm (149) and Qwen2.5-7B's rule and random arms (105, 82); the unaided arms had one in total, and the unrepaired call-cost arm 364 for Qwen3-8B.”

May combine two claims. The quote only partly states this claim.

Stated judgments: answers on <10% after five consecutive stated useless judgments (persistent); descriptive below 20 questions and for Qwen3-32B. Outcomes: pass 2/2: Llama-3.1-8B 1/266, Qwen3-8B 1/287; Qwen2.5-7B 0/15, Qwen3-32B 16/252 (descriptive).

“pass 2/2: Llama-3.1-8B 1/266, Qwen3-8B 1/287; Qwen2.5-7B 0/15, Qwen3-32B 16/252 (descriptive)”

Classified as probably not a substantial claim. May combine two claims.

Claude Haiku 4.5, call cost stated: design-label contrast CI includes or lies below 0; utility at λ=0.05 reported. Outcomes: pass: +.013 [-.023,+.048]; mean6 vs. enforced rule +.053; utility vs. none +.154, vs. enforced rule +.213.

“pass: + . 013 [ - . 023 , + . 048]; mean6 vs. enforced rule + . 053; utility vs. none + . 154, vs. enforced rule + . 213”

Classified as probably not a substantial claim. May combine two claims.

The rule fired 498 times over the six mean6 regimes for Search-R1, and the agent answered after 424 of them.

“The rule fired 498 times over the six mean6 regimes, and the agent answered after 424 of them; on a persistently failing source it ends without a well-formed answer on 12% of questions (none: 3%).”

May combine two claims.

S-H2 passed (+0.11 [+0.05,+0.18]); S-H1 failed (+0.022 [-0.003,+0.052], p=0.051).

“S-H2 passed ( + 0 . 11 [ + 0 . 05, + 0 . 18]); S-H1 failed ( + 0 . 022 [ -0 . 003, + 0 . 052], 𝑝 = 0 . 051).”

May combine two claims.

The stated cost (0.05 per call, with the budget) left the 7-8B models answering at the deadline (median action 8; 53-86% of failing-source answers at the final action) and did not change cost-adjusted success relative to the budget prompt (-0.009 to +0.012); the combination's cost-adjusted success exceeds the stated cost's for the 7-8B models (+0.039 to +0.095) and ties it for Qwen3-32B (+0.003 [-

“The stated cost (0.05 per call, with the budget) left the 7-8B models answering at the deadline (median action 8; 53-86% of failing-source answers at the final action) and did not change cost-adjusted success relative to the budget prompt ( -0 . 009 to + 0 . 012); the combination's cost-adjusted success exceeds the stated cost's for the 7-8B models ( + 0 . 039 to + 0 . 095) and ties it for Qwen3-3”

May combine two claims.

Where the agent states a judgment in its next thought, that judgment agrees with the side-channel judgment replayed at the same point as follows: Qwen2.5-7B 0.87 (κ 0.60; 99 of 160 explicit); Llama-3.1-8B 0.94 (κ 0.84; 155 of 160 explicit); Qwen3-8B 0.94 (κ 0.84; 156 of 160 explicit); Qwen3-32B 0.96 (κ 0.89; 157 of 160 explicit).

“Where the agent states a judgment in its next thought, that judgment agrees with the side-channel judgment replayed at the same point as follows: Qwen2.5-7B 0.87 ( 𝜅 0.60; 99 of 160 explicit); Llama-3.1-8B 0.94 ( 𝜅 0.84; 155 of 160 explicit); Qwen3-8B 0.94 ( 𝜅 0.84; 156 of 160 explicit); Qwen3-32B 0.96 ( 𝜅 0.89; 157 of 160 explicit).”

May combine two claims.

The budget cue's tool calls grow with the budget, whereas the rule's and the combination's barely change.

“the budget cue's calls grow with the budget, whereas the rule's and the combination's barely change.”

Classified as probably not a substantial claim.

The rule fired on 65 of 100 persistent questions.

“The rule fired on 65 of 100 persistent questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After the runs, answers that copied the template finish[answer] (the word 'answer' or no argument) were found and scored wrong.

“After the runs we found answers that copied the template finish[answer] : the word 'answer' (sometimes 'guess') or no argument, scored wrong.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Including 2,628 in the answer-now branch, the local models produced 3,834 placeholders (stated-rule and combination arms excluded); 3,254 became real answers, 545 correct.

“Including 2,628 in the answer-now branch, the local models produced 3,834 (stated-rule and combination arms excluded); 3,254 became real answers, 545 correct”

Probably not a substantial claim: may not pass S3 (load-bearing).

No decision in Table 12 changed.

“No decision in Table 12 changed”

May combine two claims.

E2 is a non-inferiority test with margin -0.05, passed by all four models.

“E2 is a non-inferiority test with margin -0 . 05, passed by all four models.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

FEVER's seven original arms had 606 placeholders, 339 of them empty answers from Llama-3.1-8B's decide arm.

“FEVER's seven original arms had 606, 339 of them empty answers from Llama-3.1-8B's decide arm.”

May combine two claims.

Stated-judgment rule (keyword reader on agent's own thought, no judgment call) > none (mean6); one-sided, Holm. Outcomes: pass 3/3 applicable: +.064/+.036/+.064 (Llama-3.1-8B, Qwen3-8B, Qwen3-32B); Qwen2.5-7B +.018, not applicable (fires on 0.05 of persistent questions).

“pass 3/3 applicable: + . 064 / + . 036 / + . 064 (Llama-3.1-8B, Qwen3-8B, Qwen3-32B); Qwen2.5-7B + . 018, not applicable (fires on 0.05 of persistent questions)”

May combine two claims.

The integration index is positive for a clock and for a hazard that rises with time, and undefined when stopping precedes four failures; both contrasts are zero in expectation for every time- or deadline-driven policy and positive for every evidence-driven policy whose threshold falls within the budget. No measure detects a threshold beyond the budget (k=7).

“The integration index is positive for a clock and for a hazard that rises with time, and undefined when stopping precedes four failures; both contrasts are zero in expectation for every timeor deadline-driven policy and positive for every evidence-driven policy whose threshold falls within the budget. No measure detects a threshold beyond the budget ( 𝑘 = 7).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The forced-answer wording and the prompt arms' 'use finish[answer] with your best guess' elicited placeholders most.

“The forced-answer wording and the prompt arms' 'use finish[answer] with your best guess' elicited them most.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Under these prompts Claude Haiku 4.5 often wrote its native tool-call markup instead of an action line, or answered in prose without the finish action.

“Under these prompts Claude Haiku 4.5 often wrote its native tool-call markup instead of an action line, or answered in prose without the finish action”

May combine two claims. The quote only partly states this claim.

Search-R1: unaided answers on <10% after five own useless judgments (persistent). Outcomes: fail: 29/165 (.18).

“fail : 29/165 (.18)”

Probably not a substantial claim: may not pass S3 (load-bearing).

All failures remain (Qwen2.5-7B's H7 at p = 0.0505).

“all failures remain (Qwen2.5-7B's H7 at 𝑝 = 0 . 0505)”

May combine two claims.

With the price stated, utility at λ=0.05 exceeds the unaided agent's by +0.154 and the enforced rule's by +0.213 (the latter counting its judgment calls).

“With the price stated, utility at 𝜆 = 0 . 05 exceeds the unaided agent's by + 0 . 154 and the enforced rule's by + 0 . 213 (the latter counting its judgment calls).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each fit uses 3,064-6,434 decisions, 5-31% of them with an earlier useful judgment.

“Each fit uses 3,064-6,434 decisions, 5-31% of them with an earlier useful judgment.”

May combine two claims.

Two LLM annotators of different families labeled all 1,844 observations and 1,675 stated judgments independently; on 1,792 and 1,672 items both labeled validly they agree at five-class κ 0.617 (S1) and 0.844 (S2), binary 0.927 and 0.903.

“On the 1,792 and 1,672 items both labeled validly they agree at five-class 𝜅 0.617 (S1) and 0.844 (S2), binary 0.927 (useful new or not) and 0.903 (says useless or not)”

May combine two claims. The quote only partly states this claim.

For Qwen2.5-7B, the stated-judgment rule fires on 5% of persistently failing questions, below the pre-registered 20% applicability threshold.

“| Qwen2.5-7B | + 0 . 018 ∗ [ + 0 . 011 , + 0 . 024] ∗ | - 0 . 027 [ - 0 . 041 , - 0 . 012] | + 0 . 086 ∗ [ + 0 . 070 , + 0 . 102] | + 0 . 027 ∗ [ + 0 . 018 , + 0 . 037] | 0.05 |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Three contrasts in the table changed significance, as did Qwen3-8B's stated-rule contrast in PP2 (Table 5).

“Three contrasts in the table changed significance, as did Qwen3-8B's stated-rule contrast in PP2 (Table 5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The realized value is the success on this source, 0.00-0.04.

“the realized value is the success on this source, 0.00-0.04”

May combine two claims.

Search-R1's judgments call 100% of failed and 89% of real observations useless.

“Its judgments call 100% of failed and 89% of real observations useless (pre-registered validity: at most 50% of real ones; the other open models' rule-arm logs on the same six regimes: 35-73%), so its own-judgment contrast ( -0 . 26 [ -0 . 58 , -0 . 06]) is not interpreted.”

May combine two claims.

Unaided agents do not rise with j, except for a partial rise in Claude Sonnet 5.

“Unaided agents (gray) do not rise with 𝑗 , except for a partial rise in Claude Sonnet 5.”

May combine two claims.

Prompt cues stay flat or fall in the 7-8B models and rise only in Qwen3-32B.

“Prompt cues (purple) stay flat or fall in the 7-8B models and rise only in Qwen3-32B.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All other outcomes are unchanged by the repair.

“All other outcomes are unchanged by the repair.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Rp4 (combination non-inferior to both components, margin -0.05) passes five of six tests and fails against the rule for Qwen3-8B (lower bound -0.054), but passes all six after answer repair (+0.008 [-0.020, +0.036] for Qwen3-8B).

“Rp4 (combination non-inferior to both components, margin -0 . 05) passes five of six tests and fails against the rule for Qwen3-8B (lower bound -0 . 054), but passes all six after answer repair ( + 0 . 008 [ -0 . 020 , + 0 . 036] for Qwen3-8B).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Replicate minus original mean6 is within ±0.023 for every model and arm.

“Replicate minus original mean6 is within ± 0 . 023 for every model and arm.”

May combine two claims.

After five consecutive useless judgments, the unaided agent answers on 41 of 202, 9 of 166, 26 of 170 and 26 of 122 questions, respectively.

“After five consecutive useless judgments, it answers on 41 of 202, 9 of 166, 26 of 170 and 26 of 122 questions, respectively.”

May combine two claims.

Switch rule minus none, mean over the six regimes: Qwen2.5-7B +0.011 [-0.005, +0.026]; Llama-3.1-8B +0.054 [+0.037, +0.071]; Qwen3-8B +0.016 [+0.005, +0.026]; Qwen3-32B +0.007 [-0.008, +0.024].

“Switch rule minus none, mean over the six regimes: Qwen2.5-7B + 0 . 011 [ -0 . 005 , + 0 . 026]; Llama-3.1-8B + 0 . 054 [ + 0 . 037 , + 0 . 071]; Qwen3-8B + 0 . 016 [ + 0 . 005 , + 0 . 026]; Qwen3-32B + 0 . 007 [ -0 . 008 , + 0 . 024].”

May combine two claims.

The pattern replicates and is not specific to 7-8B models.

“The pattern replicates and is not specific to 7-8B models”

Probably not a substantial claim: may not pass S3 (load-bearing).

All hypotheses were registered before the corresponding held-out runs.

“All hypotheses were registered before the corresponding held-out runs.”

May combine two claims.

Telling agents more changes when they stop but not what they stop on.

“Telling agents more changes when they stop but not what they stop on”

Classified as probably not a substantial claim.

All other results rest on design labels, side-channel judgments or deterministic action parsing, which had no errors in a 100-point audit.

“All other results rest on design labels, side-channel judgments or deterministic action parsing, which had no errors in a 100-point audit.”

May combine two claims.

Beliefs point in different directions across models, and actions do not follow them.

“Beliefs thus point in different directions across models, and actions do not follow them.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The amendments that define Δ followed earlier results on the same questions, so the predictions were re-tested on 300 questions never used before (fresh300).

“The amendments that define Δ followed earlier results on the same questions, so we re-test their predictions on 300 questions never used before (fresh300).”

Classified as probably not a substantial claim. May combine two claims.

Stated judgments were labeled by two LLM annotators from different model families with author adjudication, achieving binary κ 0.90.

“Stated judgments have to be read from free text, so two LLM annotators from different model families labeled them, with author adjudication (binary κ 0.90; Appendix B.2).”

Classified as probably not a substantial claim. May combine two claims.

The time-matched contrast Δ is zero for stopping by time or the deadline, positive for stopping on the useless run, and negative when stopping follows earlier useful evidence.

“Hence Δ is zero for stopping by time or the deadline, positive for stopping on the useless run, and negative when stopping follows earlier useful evidence (Figure 2).”

• Also stated in: § source recovers. · p. 2 · plain

  “We test this prediction with the time-matched contrast Δ , which is positive for an agent that integrates, zero for one that stops by the clock or the deadline, and negative for one whose stops follow useful evidence instead.”

May combine two claims.

Agents judge correctly but do not act on their judgments.

“Agents judge correctly but do not act on their judgments”

• AssumptionRests on an assumption: Tools often fail silently.

Classified as probably not a substantial claim. May combine two claims.

Every threshold was set on 100 development questions; confirmatory results use a disjoint set of 300 (test300) and the FEVER claims.

“Every threshold was set on 100 development questions; confirmatory results use a disjoint set of 300 (test300) and the FEVER claims.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Search-augmented models over-search.

“Search-augmented models over-search [Liu et al., 2026a, Xie et al., 2026].”

• cites Liu et al., 2026aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xie et al., 2026 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The gap between judging and doing also appears in bandit tasks, where models state the right strategy but act greedily.

“The gap also appears in bandit tasks, where models state the right strategy but act greedily [Schmied et al., 2026].”

• cites Schmied et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Qwen2.5-7B result was recorded in a results file before its test runs but was not entered in the document as the protocol required, which we count as a deviation.

“The Qwen2.5-7B result was recorded in a results file before its test runs but was not entered in the document as the protocol required, which we count as a deviation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The document was not deposited with a third-party registry.

“The document was not deposited with a third-party registry.”

Probably not a substantial claim: may not pass S2 (checkable).

The missing piece is integration.

“The missing piece is integration.”

Classified as probably not a substantial claim. May combine two claims.

Differences are paired by question, with 95% intervals from 2,000 bootstrap resamples (clustered by question for I); tests across models are Holm-corrected; non-inferiority margins are -0.05.

“Differences are paired by question, with 95% intervals from 2,000 bootstrap resamples (clustered by question for 𝐼 ); tests across models are Holm-corrected; non-inferiority margins are -0 . 05.”

May combine two claims.

An agent's stated judgment of its evidence does not guarantee that it acts on it, so the two must be measured separately.

“An agent's stated judgment of its evidence does not guarantee that it acts on it, so the two must be measured separately.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The pre-registration and its amendments were kept as one append-only document.

“The pre-registration and its amendments were kept as one appendonly document”

Classified as probably not a substantial claim. May combine two claims.

The paper's results argue for making the decision to abandon a source explicit and evaluating it, rather than for abandoning tools.

“Our results therefore argue for making the decision to abandon a source explicit and evaluating it, rather than for abandoning tools.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

When recovery comes later, the enforced rule can give up just before it, so the threshold would need tuning for real sources.

“When recovery comes later, the rule can give up just before it (Appendix D.5), so the threshold would need tuning for real sources.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Judgments are elicited on a copy of the conversation and beliefs are self-reports, so they are compared only within a model.

“Judgments are elicited on a copy of the conversation and beliefs are self-reports, so we compare them only within a model.”

Classified as probably not a substantial claim.

The stop rule (withdraw the intervention claim if H1 failed for two of three 7-8B models) did not trigger.

“The stop rule (withdraw the intervention claim if H1 failed for two of three 7-8B models) did not trigger.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Models largely know what they know.

“Models largely know what they know [Kadavath et al., 2022].”

• cites Kadavath et al., 2022 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Answering from memory is not automatically safe, since parametric knowledge can be stale or wrong.

“Answering from memory is not automatically safe, since parametric knowledge can be stale or wrong.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Run-to-run variation arises when an intervention changes vLLM's batching.

“so run-to-run variation arises when an intervention changes vLLM's batching.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stated judgments were labeled by LLM annotators with author adjudication rather than by independent human annotators.

“Stated judgments were labeled by LLM annotators with author adjudication rather than by independent human annotators.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Making the judgment explicit reduces the gap between judging and doing.

“and making the judgment explicit reduces it [Cheng et al., 2026].”

• cites Cheng et al., 2026 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Holm correction is applied across models within each hypothesis, not across the many hypotheses in Tables 4 and 5.

“Holm correction is applied across models within each hypothesis, not across the many hypotheses in Tables 4 and 5”

Probably not a substantial claim: may not pass S4 (based). May combine two claims. The quote only partly states this claim.

The enforced rule raises success because it acts on an accurate signal.

“because it acts on an accurate signal”

Probably not a substantial claim: may not pass S3 (load-bearing).

A gap between judging and doing has been shown for the decision to call a tool at all.

“A gap between judging and doing has nonetheless been shown for the decision to call a tool at all [Li et al., 2026]”

• cites Li et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

All thresholds were set on the 100 development questions.

“All thresholds were set on the 100 development questions.”

Classified as probably not a substantial claim.

Deepresearch agents keep browsing results that have turned out irrelevant.

“deepresearch agents keep browsing results that have turned out irrelevant, which Zhang et al. [2026b] attribute to distorted judgment.”

• cites Zhang et al., 2026bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Each amendment was frozen before the runs it governs.

“Each amendment was frozen before the runs it governs”

Probably not a substantial claim: may not pass S3 (load-bearing).

All numbers in this appendix are computed by one script from the run logs.

“All numbers in this appendix are computed by one script from the run logs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Amendment 12 was frozen before any of its runs.

“Amendment 12 was frozen before any of its runs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Claude Sonnet 5's per-step judgments were not saved.

“its per-step judgments were not saved (Appendix B.4)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both amendment 19 and amendment 20 were frozen before their test runs.

“Both were frozen before their test runs (Table 5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some arms were pre-registered.

“pre-registered for some arms, Appendix A”

Probably not a substantial claim: may not pass S3 (load-bearing).

HotpotQA results use test300 with the 8-action budget (Claude Sonnet 5: its first 100 questions).

“HotpotQA results use test300 with the 8-action budget (Claude Sonnet 5: its first 100 questions)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen3-32B, Claude Haiku 4.5 and Claude Sonnet 5 were not run on FEVER.

“Qwen3-32B, Claude Haiku 4.5 and Claude Sonnet 5 were not run on FEVER.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No plausible regime was run for FEVER.

“for which no plausible regime was run”

Probably not a substantial claim: may not pass S3 (load-bearing).

The first author adjudicated all 59 disagreements that change a consecutive-useless count.

“The first author adjudicated all 59 disagreements that change a consecutive-useless count.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

These analyses are exploratory.

“These analyses are exploratory.”

Classified as probably not a substantial claim. May combine two claims.

The rule run's per-step side-channel judgments were not saved for five of its six regimes, so the pre-registered share of 'useless' judgments and its own-judgment measures cannot be computed.

“The rule run's per-step side-channel judgments were not saved for five of its six regimes, so the pre-registered share of 'useless' judgments and its own-judgment measures cannot be computed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Paired contrasts in mean6 between arms use 300 paired questions on HotpotQA (Claude Sonnet 5: 100) and 292 on FEVER.

“𝑛 = 300 paired questions on HotpotQA (Claude Sonnet 5: 100) and 292 on FEVER”

Probably not a substantial claim: may not pass S3 (load-bearing).

Unless a column says repaired, numbers come from the raw, pre-registered analysis.

“Unless a column says repaired , numbers come from the raw, pre-registered analysis”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The belief probe analysis is descriptive, not pre-registered.

“this analysis is descriptive, not pre-registered”

May combine two claims.

The earlier unaided and enforced-rule runs lacked the format fixes and had 2-3% format failures and a further 2% truncated generations on these questions, which favors the new conditions, most of all the call-cost condition with almost no format failures.

“The earlier unaided and enforced-rule runs lacked them and had 2-3% format failures and a further 2% truncated generations on these questions, which favors the new conditions, most of all the call-cost condition with almost no format failures.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Contrasts between answering arms could move either way, and stopping measures are unaffected.

“Contrasts between answering arms could move either way, and stopping measures are unaffected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ideal observer E[Vc|k] after k results that all failed is a reference rather than an optimum.

“The ideal observer 𝐸 [ 𝑉𝑐 | 𝑘 ] after 𝑘 results that all failed is a reference rather than an optimum.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The one-sided p is the share of resampled differences ≤ 0, unadjusted for multiplicity.

“the one-sided 𝑝 is the share of resampled differences ≤ 0, unadjusted for multiplicity”

Classified as probably not a substantial claim.

A negative Δ in the hazard model could come from answering on sufficient evidence rather than from ignoring useless evidence.

“Δ compares histories with and without an earlier result judged useful, so a negative Δ could come from answering on sufficient evidence rather than from ignoring useless evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Amendments 15-17 were frozen before any of their runs.

“Amendments 15-17 were frozen before any of their runs (Table 5).”

Classified as probably not a substantial claim. May combine two claims.

Held-out questions were taken in order of released question manifest and first one selected where (1) Claude Haiku 4.5 without intervention exhausts budget in persistent regime, (2) on same question rule arm's side-channel judgments are USELESS five times in a row and rule fires, and (3) rule arm's answer is correct. No other candidate was inspected.

“Held-out questions were taken in the order of the released question manifest and the first one was selected where (1) Claude Haiku 4.5 without intervention exhausts its budget in the persistent regime, (2) on the same question the rule arm's side-channel judgments are USELESS five times in a row and the rule fires, and (3) the rule arm's answer is correct. No other candidate was inspected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The comparison is exploratory and specific to sources that, once failed five times, do not recover.

“The comparison is exploratory and specific to sources that, once failed five times, do not recover.”

May combine two claims.

Placeholders occurred almost only in arms that stop and answer, so H1 and H2 were conservative.

“they occurred almost only in arms that stop and answer, so H1 and H2 were conservative.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Agents obey the rule's finish instruction at 89-98% of decision points.

“which the agents obey at 89-98% of decision points”

Probably not a substantial claim: may not pass S3 (load-bearing).

FEVER results use all 292 claims.

“FEVER results use all 292 claims”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A local Qwen3-32B annotator labeled the 327 adjudicated stated-judgment items with five-class κ 0.842 [0.766, 0.910] and binary κ 0.874 [0.798, 0.939].

“A local Qwen3-32B annotator (nonthinking, temperature 0, the rubric of the LLM annotators) labeled the 327 adjudicated statedjudgment items with five-class 𝜅 0.842 [0.766, 0.910] and binary (useless or not) 𝜅 0.874 [0.798, 0.939].”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the raw analysis, a template-placeholder answer counts as wrong.

“in which a template-placeholder answer counts as wrong”

Probably not a substantial claim: may not pass S3 (load-bearing).

Intervals are 95% percentile intervals over 2,000 bootstrap resamples of questions (for I, whole questions are resampled with all their decision points).

“Intervals are 95% percentile intervals over 2,000 bootstrap resamples of questions (for 𝐼 , whole questions are resampled with all their decision points)”

Probably not a substantial claim: may not pass S3 (load-bearing).

No model was trained or fine-tuned.

“No model was trained or fine-tuned.”

Classified as probably not a substantial claim.

Before any run on them, we sampled 300 new questions from the HotpotQA distractor development set with a new random seed, disjoint from every question used before (pilot, development and test; the list is in the code release).

“Before any run on them, we sampled 300 new questions from the HotpotQA distractor development set with a new random seed, disjoint from every question used before (pilot, development and test; the list is in the code release).”

Classified as probably not a substantial claim.

Δ was the primary endpoint.

“Δ was the primary endpoint.”

Classified as probably not a substantial claim. May combine two claims.

Amendment 10 repeated the none, budget, rule and combination arms on the three 7-8B models with new sampling seeds (Qwen3) and a new draw of the failed observations (all models), keeping questions, regimes and prompts fixed; it was frozen before the replicate runs.

“Amendment 10 repeated the none, budget, rule and combination arms on the three 7-8B models with new sampling seeds (Qwen3) and a new draw of the failed observations (all models), keeping questions, regimes and prompts fixed; it was frozen before the replicate runs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Amendment 13 was frozen before its runs; all runs are local.

“Amendment 13 was frozen before its runs; all runs are local.”

May combine two claims.

By the pre-registered interpretation, Qwen2.5-7B and Llama-3.1-8B do not abandon the primary source, and Qwen3-8B and Qwen3-32B switch.

“By the pre-registered interpretation, a model 'switches' if its share switched by the fourth action is at least 0.5, and otherwise 'does not abandon' if the contrast's interval includes or lies below 0. Qwen2.5-7B and Llama-3.1-8B therefore do not abandon, and Qwen38B and Qwen3-32B switch.”