HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Probably not a substantial claim: may not pass S3 (load-bearing).

TextGrad, Trace, and LLM-AutoDiff optimize arbitrary text components in chained workflows by back-propagating textual feedback using LLMs.

“TextGrad [34], Trace [7], and LLM-AutoDiff [32] optimize arbitrary text components in chained workflows by back-propagating textual feedback using LLMs.”

• cites [34] [34] M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou. Textgrad: Automatic "differentiation" via text, 2024. URL https://arxiv.org/abs/2406.07496 . - sourceSupported
• cites [7] [7] C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the next AutoDiff: Generative optimization with rich feedback, execution traces, and LLMs. Advances in Neural Information Processing Systems (NeurIPS) , 2024. URL https://arxiv.org/abs/2406.16218 . - sourceDisputed

  Section 5.5 explicitly distinguishes the methods: TextGrad propagates text feedback, whereas Trace propagates minimal subgraphs. Section 1.1 likewise says Trace propagates execution traces rather than gradients. Although Trace can optimize prompts in workflows, the mechanism attributed to Trace here is contrary to the paper. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites [32] [32] L. Yin and Z. Wang. LLM-AutoDiff: Auto-differentiate any LLM workflow. arXiv preprint arXiv:2501.16673 , 2025. URL https://arxiv.org/abs/2501.16673 . <!-- image --> IscaLe - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prompt optimization methods such as DSPy optimize prompts while holding the surrounding program fixed.

“Prompt optimization methods such as DSPy [10] optimize prompts while holding the surrounding program fixed.”

• cites [10] [10] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. In International Conference on Learning Representations (ICLR) , 2024. URL https://arxiv.org/abs/2310.03714 . - sourceDisputed

  The DSPy Compiler (p. 6) describes optimizing prompt parameters such as instructions and demonstrations. But Stage 3 (p. 7) explicitly says the compiler also supports modifying program control flow, including replacing a program with an ensemble. DSPy therefore does not necessarily hold the surrounding program fixed. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Other systems in this regime include Agentic Harness Engineering and HarnessX.

“Other systems in this regime include Agentic Harness Engineering [13] and HarnessX [5].”

• cites [13] [13] J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, Z. Xi, X. Huang, H. Yan, Z. Han, T. Gui, and Y.-G. Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses, 2026. URL https://arxiv.org/abs/2604.25850 . - sourceSupported
• cites [5] [5] T. Chen, S. Lu, K. Zhao, W. Meng, H. Teng, T. Li, C. Li, X. Liu, J. Liang, Z. Zhang, Y. Xie, H. Qu, K. Shao, and J. Luan. HarnessX: A composable, adaptive, and evolvable agent harness foundry, 2026. URL https://arxiv.org/abs/2606.14249 . -Supported

Probably not a substantial claim: may not pass S3 (load-bearing).

Existing approaches to harness optimization differ along two axes: which parts of the harness they may modify, and the role assigned to the LLM during search.

“Existing approaches to harness optimization differ along two axes: which parts of the harness they may modify, and the role assigned to the LLM during search.”

• cites [10] [10] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. In International Conference on Learning Representations (ICLR) , 2024. URL https://arxiv.org/abs/2310.03714 . - sourcePartly supported

  Related Work (p. 3) distinguishes prompt optimization for a single LM call from optimization of arbitrary pipelines, and mentions both LM feedback and model-selection methods. The DSPy Compiler (pp. 6–7) describes modifying instructions, demonstrations, weights, and control flow. These passages support differences in what is optimized and how, but they do not present existing approaches as differing along the claimed two-axis framework or systematically compare the LLM’s role during search. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [31] [31] H. Ye, X. He, V. Arak, H. Dong, and G. Song. Meta context engineering via agentic skill evolution, 2026. URL https://arxiv.org/abs/2601.21557 . - sourcePartly supported

  Section 1 (Introduction; page unavailable) distinguishes existing context-engineering approaches by what they optimize, while §2.2 distinguishes ways LLMs participate in evolutionary optimization. The paper does not explicitly classify existing harness-optimization approaches along the two stated axes, and its discussion of LLM roles concerns evolutionary computation more broadly. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [1] [1] L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations , 2026. URL https://openreview.net/forum?id=RQm2KQTM5r . - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

ADAS searches over agent designs represented as code.

“ADAS [8] searches over agent designs represented as code.”

• cites [8] [8] S. Hu, C. Lu, and J. Clune. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=t9U3LW7JVX . - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

AFlow performs tree search over code-defined workflows.

“AFlow [37] performs tree search over code-defined workflows.”

• cites [37] [37] J. Zhang, J. Xiang, Z. Yu, F. Teng, X.-H. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, et al. AFlow: Automating agentic workflow generation. In International Conference on Learning Representations (ICLR) , 2025. URL https://arxiv.org/abs/2410.10762 . - sourceSupported

May combine two claims.

Methods are today evaluated with their own target agents, seeds, budgets, disclosure policies, and scoring protocols, so their results conflate the optimizer model, the coding harness it acts through, the target agent, and the protocol.

“Methods are today evaluated with their own target agents, seeds, budgets, disclosure policies, and scoring protocols, so their results conflate the optimizer model, the coding harness it acts through, the target agent, and the protocol.”

• cites [23] [23] V. Ursekar, A. Shanker, V. Chatrath, Y. Xue, and S. M. Denton. VeRO: A harness for agents to optimize agents. In Forty-third International Conference on Machine Learning , 2026. URL https://openreview. net/forum?id=zQzmwG2Nue . - sourcePartly supported

  Section 3.2, “A Protocol for Benchmarking Coding Agents as Agent Optimizers” (page unavailable), says meaningful comparisons require identical resource and environment conditions. The Protocol section (page unavailable) also distinguishes the coding-agent scaffold from the optimizer model. But the paper does not establish the broader claim that methods today each use different target agents, seeds, disclosure policies, and scoring protocols, or that their reported results consequently conflate all four named factors. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [12] [12] Y. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn. Meta-harness: End-to-end optimization of model harnesses, 2026. URL https://arxiv.org/abs/2603.28052 . - sourcePartly supported

  Section 4.1 (p. 6) says the paper holds the proposer configuration, evaluation budget, and test-set access constant for a fair comparison, supporting the concern that these choices affect comparisons. Section 4.3 (p. 9) also reports difficulty reproducing another agent’s leaderboard score from its public code. The paper does not establish the broader claim that methods generally use different seeds, disclosure policies, or scoring protocols, or that reported results conflate all four named components. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

May combine two claims.

A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit.

“A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit.”

• Also stated in: § ( RQ3 ). · p. 3 · plain

  “A trusted execution environment, building on VERO [23], enforces access and target-evaluation budgets, isolates held-out state, meters resource use, and versions every candidate for audit (Figure 1).”

• cites [23] [23] V. Ursekar, A. Shanker, V. Chatrath, Y. Xue, and S. M. Denton. VeRO: A harness for agents to optimize agents. In Forty-third International Conference on Machine Learning , 2026. URL https://openreview. net/forum?id=zQzmwG2Nue . - sourcePartly supported

  The Permission control and Experiment Database sections describe enforced evaluation restrictions and a gated runner that counts evaluation calls. The Git Worktree section describes an immutable commit history that makes changes auditable. The paper does not attribute these functions to a trusted execution environment, and its A.4 Limitations section says the budget does not measure broader resource use such as tokens or API costs. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

GEPA optimizes textual artifacts by using reflective feedback to guide an evolutionary process.

“GEPA [1] optimizes textual artifacts by using reflective feedback to guide an evolutionary process.”

• cites [1] [1] L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations , 2026. URL https://openreview.net/forum?id=RQm2KQTM5r . - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

STOP recursively improves a scaffolding program.

“STOP [35] recursively improves a scaffolding program.”

• cites [35] [35] E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation. In First Conference on Language Modeling , 2024. URL https://openreview.net/forum? id=46Zgqo4QIU . - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Meta Context Engineering searches over skills and context artifacts in a bi-level procedure.

“Meta Context Engineering [31] searches over skills and context artifacts in a bi-level procedure.”

• cites [31] [31] H. Ye, X. He, V. Arak, H. Dong, and G. Song. Meta context engineering via agentic skill evolution, 2026. URL https://arxiv.org/abs/2601.21557 . - sourceSupported

May combine two claims.

The strongest configuration captures roughly two thirds of the available OfficeQA headroom and half of the BrowseComp-Plus headroom.

“The strongest configuration captures roughly two thirds of the available OfficeQA headroom and half of the BrowseComp-Plus headroom”

May combine two claims.

Detailed failure-trace inspection is rarely used and is not positively associated with gain.

“Instrumented search trajectories show that broader intervention is associated with greater held-out gain, whereas detailed failure-trace inspection is rarely used and is not positively associated with gain (Section 5.3).”

May combine two claims.

On GAIA, no harness holds its rank across models; the best harness depends on the model.

“On the one task where the harness really varies, the best harness depends on the model.”

• Also stated in: § 5.3 Where current optimizers fall short · p. 10 · plain

  “the lines cross, so no harness holds its rank across models.”

• Also stated in: § 5.4 The effect of the optimizer's own harness · p. 11 · plain

  “The harnesses do not hold their rank across models”

May combine two claims.

Three of the four seeds are competent but naive; the OfficeQA seed is a ∼130-line agent built on the OpenAI API, with three tools, a 24-turn loop, and a generic system prompt. GAIA's is a non-functional stub.

“Three of the four are competent but naive; the OfficeQA seed, for example, is a ∼ 130-line agent built on the OpenAI API, with three tools, a 24-turn loop, and a generic system prompt. GAIA's is a non-functional stub.”

May combine two claims.

On GAIA, both GPT models are four to five resolution bands (or +0.179 and +0.131) better under codex than their best shared harness, while the two Claudes and Kimi sit within a band or two of zero.

“Both GPT models are far better under codex ( + 0.179 and + 0.131 ); the two Claudes and Kimi sit within a band or two of zero either way.”

• Also stated in: § 5.4 The effect of the optimizer's own harness · p. 11 · plain

  “both GPT models are four to five resolution bands better under codex , while the two Claudes and Kimi sit within a band or two of zero.”

May combine two claims.

Across 5 Claude Opus releases, gain ranges from +0.37 to +0.59; gain is non-monotonic, but the first-to-last spread exceeds the task's resolution band.

“Across 5 Claude Opus releases, gain ranges from +0.37 to +0.59; gain is non-monotonic, but the first-to-last spread exceeds the task's resolution band.”

May combine two claims.

Across 5 GPT releases, gain rises monotonically from +0.03 to +0.49, with three of its four steps exceeding the task's resolution band.

“Across 5 GPT releases, gain rises monotonically from +0.03 to +0.49, with three of its four steps exceeding the task's resolution band.”

May combine two claims.

Both exceed the task resolution bands, although the harness contrast does so narrowly.

“Both exceed the task resolution bands, although the harness contrast does so narrowly.”

May combine two claims.

Differences among intermediate configurations are often smaller than their round-to-round variation, supporting tiers rather than a complete ranking.

“Differences among intermediate configurations are often smaller than their round-to-round variation, supporting tiers rather than a complete ranking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On BrowseComp-Plus, gpt-5.6-sol with the opencode harness had a mean Levers value of 0.50 (observed range 0.25-0.75).

“| gpt-5.6-sol | opencode | 0.09 ( 0.05 - 0.14 ) | 0.50 ( 0.25 - 0.75 ) | 7.59 ( 5.18 - 10.00 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, kimi-k3 with the opencode harness used a mean of 3.00M target tokens (observed range 2.21-3.79M).

“| kimi-k3 | opencode | 0.41 ( 0.23 - 0.59 ) | 0.94 ( 0.88 - 1.00 ) | 3.00 ( 2.21 - 3.79 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On BrowseComp-Plus, gpt-5.6-sol with the codex harness used a mean of 4.04M target tokens (observed range 3.33-4.76M).

“| gpt-5.6-sol | codex | 0.03 ( 0.02 - 0.03 ) | 0.69 ( 0.62 - 0.75 ) | 4.04 ( 3.33 - 4.76 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, claude-opus-5 with the opencode harness had a mean Levers value of 1.00 (observed range 1.00-1.00).

“| claude-opus-5 | opencode | 0.63 ( 0.60 - 0.67 ) | 1.00 ( 1.00 - 1.00 ) | 3.77 ( 3.36 - 4.18 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, claude-opus-5 with the opencode harness used a mean of 3.77M target tokens (observed range 3.36-4.18M).

“| claude-opus-5 | opencode | 0.63 ( 0.60 - 0.67 ) | 1.00 ( 1.00 - 1.00 ) | 3.77 ( 3.36 - 4.18 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, gpt-5.6-terra with the codex harness used a mean of 1.26M target tokens (observed range 1.15-1.37M).

“| gpt-5.6-terra gpt-5.6-terra | codex opencode | 0.07 ( - 0.04 - 0.17 ) | 0.44 ( 0.38 - 0.50 ) | 1.26 ( 1.15 - 1.37 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On BrowseComp-Plus, claude-opus-5 with the opencode harness used a mean of 12.50M target tokens (observed range 11.13-13.88M).

“| claude-opus-5 | opencode | 0.48 ( 0.42 - 0.55 ) | 1.00 ( 1.00 - 1.00 ) | ) 12.50 ( 11.13 - 13.88 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On BrowseComp-Plus, claude-opus-5 with the claude-code harness had a mean Levers value of 1.00 (observed range 1.00-1.00).

“| claude-opus-5 | claude-code | 0.41 ( 0.38 - 0.44 ) | 1.00 ( 1.00 - 1.00 ) | 11.78 ( 9.95 - 13.61 |”

May combine two claims.

In HARNESSOPT-BENCH, the case allowance constrains optimizer search while the evaluation call cap does not.

“Case passes, not evaluation calls, bind.”

• Also stated in: § 5.3 Where current optimizers fall short · p. 10 · plain

  “Thus the case allowance constrains search, whereas the call cap does not.”

May combine two claims. The quote only partly states this claim.

On the balanced scope (15 contestant rows over 3 tasks on opencode alone), the model effect in normalized-gain units is +0.228 for claude-opus-5 (tier 1), +0.029 for claude-sonnet-5 (tier 2), -0.014 for kimi-k3 (tier 2), -0.069 for gpt-5.6-sol (tier 2), and -0.174 for gpt-5.6-terra (tier 3).

“| claude-opus-5 | + 0.228 | 1 |”

• Also stated in: § C Model effects · p. 20 · plain

  “| claude-sonnet-5 | + 0.029 | 2 |”

• Also stated in: § C Model effects · p. 20 · plain

  “| gpt-5.6-terra | - 0.174 | 3 |”

• AssumptionThe authors state this as an assumption: The model effect can be estimated by an additive fit over task and model.

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, claude-sonnet-5 with the opencode harness used a mean of 2.55M target tokens (observed range 2.31-2.79M).

“| claude-sonnet-5 | opencode | 0.51 ( 0.46 - 0.56 ) | 0.94 ( 0.88 - 1.00 ) | ) 2.55 ( 2.31 - 2.79 ) |”

May combine two claims.

Across the 20 model-task pairs evaluated under both conditions, the shared harness wins 11, the native harness wins 9, and 0 are tied.

“Across the 20 model-task pairs evaluated under both conditions, the shared harness wins 11, the native harness wins 9, and 0 are tied.”

May combine two claims.

One configuration touched three quarters of the levers and made seven edits but shipped the original seed.

“one configuration touched three quarters of the levers and made seven edits but shipped the original seed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, claude-opus-5 with the claude-code harness used a mean of 9.97M target tokens (observed range 3.05-16.89M).

“| claude-opus-5 | claude-code | 0.59 ( 0.54 - 0.64 ) | 1.00 ( 1.00 - 1.00 ) | 9.97 ( 3.05 - 16.89 ) |”

May combine two claims.

The median optimizer uses 8 evaluation calls (4%) but 82% of its case allowance, and 55 of 100 cells exhaust at least one partition's case budget.

“The median optimizer uses 8 calls (4%) but 82% of its case allowance, and 55 of 100 cells exhaust at least one partition's case budget.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, kimi-k3 with the kimi-cli harness had a mean Levers value of 1.00 (observed range 1.00-1.00).

“| kimi-k3 | kimi-cli | 0.59 ( 0.59 - 0.60 ) | 1.00 ( 1.00 - 1.00 ) | 2.77 ( 2.35 - 3.18 ) |”

The quote only partly states this claim.

On OfficeQA, gpt-5.6-terra with the opencode harness achieved a mean normalized gain of 0.14 (observed range 0.00-0.27).

“| | | 0.14 ( 0.00 - 0.27 | | |”

May combine two claims.

In 11 of the 20 model-task pairs, the difference between shared and native harness exceeds the task's resolution band, but the direction varies by model and task.

“In 11 pairs, the difference exceeds the task's resolution band, but the direction varies by model and task.”

May combine two claims.

Visible validation scores are optimistic: for most cells, the submitted candidate's test score is lower than the best validation score observed during search.

“Visible validation scores are optimistic.”

• Also stated in: § 5.3 Where current optimizers fall short · p. 10 · plain

  “Most cells in Figure 13 fall below the identity line: the submitted candidate's test score is lower than the best validation score observed during search.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On BrowseComp-Plus, claude-opus-5 with the opencode harness had a mean Levers value of 1.00 (observed range 1.00-1.00).

“| claude-opus-5 | opencode | 0.48 ( 0.42 - 0.55 ) | 1.00 ( 1.00 - 1.00 ) | ) 12.50 ( 11.13 - 13.88 ) |”

May combine two claims.

Optimizers rely mainly on per-case score summaries: detailed trace spans were requested only 16 times by 7 of the 111 cells.

“Optimizers rely mainly on per-case score summaries: detailed trace spans were requested only 16 times by 7 of the 111 cells.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, claude-opus-5 with the claude-code harness had a mean Levers value of 1.00 (observed range 1.00-1.00).

“| claude-opus-5 | claude-code | 0.59 ( 0.54 - 0.64 ) | 1.00 ( 1.00 - 1.00 ) | 9.97 ( 3.05 - 16.89 ) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

On OfficeQA, kimi-k3 with the kimi-cli harness used a mean of 2.77M target tokens (observed range 2.35-3.18M).

“| kimi-k3 | kimi-cli | 0.59 ( 0.59 - 0.60 ) | 1.00 ( 1.00 - 1.00 ) | 2.77 ( 2.35 - 3.18 ) |”

May combine two claims.

The seed agent's held-out baseline is 0.341 on OfficeQA, 0.462 on BrowseComp-Plus, 0.241 on Terminal-Bench, and 0.000 on GAIA.

“| OfficeQA | | 0.341 | 0.727 | 0.505 | 0.713 | 0 . 734 | 0.687 |”

• Also stated in: § A Full optimizer results · p. 17 · plain

  “Terminal-Bench resolution band ± 0.054 (normalized gain), baseline 0.241”

• Also stated in: § B The suite · p. 19 · plain

  “| GAIA § gpt-5.4-mini | 33/66/66 | 0.000 | 0.469 ± 0.023 | 0.207 ± 0.013 | 0 . 508 ± 0.029 | 0.172 ± 0.022 | 0.202 ± 0.005 |”

Classified as probably not a substantial claim. May combine two claims.

The optimizer model's inference is metered but left uncapped, so these results estimate what is achievable when the optimizer model's reasoning is not the scarce resource.

“The optimizer model's inference is metered but left uncapped, so these results estimate what is achievable when the optimizer model's reasoning is not the scarce resource.”

Classified as probably not a substantial claim. May combine two claims.

Raw improvement may not be comparable across tasks whose scoring scales differ, so wherever tasks are compared we use normalized gain.

“Raw improvement may not be comparable across tasks whose scoring scales differ, so wherever tasks are compared we use normalized gain”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

HARNESSOPT-BENCH similarly requires coding agents to navigate the target system codebase and iterate on environmental feedback; it differs in what the optimization target is (i.e. a harness), the noisiness of evaluation feedback, and the types of bounds imposed on its search.

“HARNESSOPT-BENCH similarly requires coding agents to navigate the target system codebase and iterate on environmental feedback; it differs in what the optimization target is (i.e. a harness), the noisiness of evaluation feedback, and the types of bounds imposed on its search.”

Classified as probably not a substantial claim. May combine two claims.

As the systems being optimized grow more complex, evaluating them only gets more expensive, so the second regime is the one that increasingly matters.

“As the systems being optimized grow more complex, evaluating them only gets more expensive, so the second regime is the one that increasingly matters.”

Probably not a substantial claim: may not pass S4 (based).

Harness optimization is a constrained, stochastic program-optimization problem.

“Harness optimization is a constrained, stochastic program-optimization problem.”

Classified as probably not a substantial claim. May combine two claims.

Scoring follows the protocol of Section 3.2: each held-out evaluation is the mean of three attempts per test case, matching the K = 3 pooling behind each task's pinned baseline.

“Scoring follows the protocol of Section 3.2: each held-out evaluation is the mean of three attempts per test case, matching the K = 3 pooling behind each task's pinned baseline.”

May combine two claims.

These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

“These results establish harness optimization as a measurable and discriminative capability with large space for improvement.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Separating them requires three conditions: the target model, environment, and verifier held fixed; the final evaluation held out throughout search, so improvement reflects generalization rather than fit to a visible score; and a trusted execution boundary that enforces the budget, blocks access to held-out state, and preserves every candidate for audit.

“Separating them requires three conditions: the target model, environment, and verifier held fixed; the final evaluation held out throughout search, so improvement reflects generalization rather than fit to a visible score; and a trusted execution boundary that enforces the budget, blocks access to held-out state, and preserves every candidate for audit.”

Classified as probably not a substantial claim.

HARNESSOPT-BENCH is complementary: it fixes the optimization problem and the evaluation protocol so that optimizer models, harnesses, and search algorithms can be compared on common ground.

“HARNESSOPT-BENCH is complementary: it fixes the optimization problem and the evaluation protocol so that optimizer models, harnesses, and search algorithms can be compared on common ground.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Harness optimization is long-horizon, plays out over a diverse and complex tool ecosystem rather than a narrow action space, and requires reasoning-driven interpretation of a stochastic system in pursuit of a measured metric rather than production of a single correct step.

“Independent of its place in that self-improvement loop, harness optimization is also long-horizon, plays out over a diverse and complex tool ecosystem rather than a narrow action space, and requires reasoning-driven interpretation of a stochastic system in pursuit of a measured metric rather than production of a single correct step.”

Classified as probably not a substantial claim. May combine two claims.

We estimate evaluation noise by scoring the same candidate twice on the same cases and carry the median discrepancy to the K = 3 normalized-gain scale used for held-out scoring. The resulting task-specific resolution band is a descriptive threshold: differences smaller than the band are treated as unresolved, not as formal significance-test results.

“We estimate evaluation noise by scoring the same candidate twice on the same cases and carry the median discrepancy to the K = 3 normalized-gain scale used for held-out scoring. The resulting task-specific resolution band is a descriptive threshold: differences smaller than the band are treated as unresolved, not as formal significance-test results.”

May combine two claims.

The suite comprises 4 downstream tasks with pinned seeds, fixed and non-overlapping development, validation, and test splits, graded disclosure, and recorded baselines for both the seed and off-the-shelf harnesses.

“The suite comprises 4 downstream tasks with pinned seeds, fixed and non-overlapping development, validation, and test splits, graded disclosure, and recorded baselines for both the seed and off-the-shelf harnesses.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Where evaluation is cheap, selection over many candidates substitutes for reasoning about any one of them; where it is expensive and noisy, reasoning from prior evidence becomes the capability itself.

“Where evaluation is cheap, selection over many candidates substitutes for reasoning about any one of them; where it is expensive and noisy, reasoning from prior evidence becomes the capability itself.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each optimizer configuration is run twice; we report ranges in Table 2.

“Each optimizer configuration is run twice; we report ranges in Table 2.”

Classified as probably not a substantial claim. May combine two claims.

Because repeated scores can differ, we summarize this evaluation noise with a task-specific resolution band; smaller differences are treated as unresolved.

“Because repeated scores can differ, we summarize this evaluation noise with a task-specific resolution band (Section 4); smaller differences are treated as unresolved.”

May combine two claims.

Trajectory-derived measures characterize how optimizers search; they are not independent measures of held-out capability.

“Trajectory-derived measures characterize how optimizers search; they are not independent measures of held-out capability.”

Classified as probably not a substantial claim. May combine two claims.

Each optimizer is run in an isolated sandbox with evaluation results, task data, and the target harness in the filesystem. Similarly, each target agent rollout is performed in an ephemeral sandbox to control for noise introduced by environmental drift.

“Each optimizer is run in an isolated sandbox with evaluation results, task data, and the target harness in the filesystem. Similarly, each target agent rollout is performed in an ephemeral sandbox to control for noise introduced by environmental drift.”

Probably not a substantial claim: may not pass S2 (checkable).

A task that combines all three properties is a demanding test of frontier capability in its own right.

“Each property is already the target of separate benchmarks; a task that combines all three is a demanding test of frontier capability in its own right.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Normalized gain is defined within a task; normalization places tasks in common headroom units but does not remove systematic differences among them.

“Normalized gain is defined within a task; normalization places tasks in common headroom units but does not remove systematic differences among them.”

May combine two claims.

Which role a domain calls for depends on the cost of one reliable evaluation, and that cost is what makes harness optimization a test of more than coding ability.

“Which role a domain calls for depends on the cost of one reliable evaluation, and that cost is what makes harness optimization a test of more than coding ability.”

Classified as probably not a substantial claim. May combine two claims.

The test partition is inaccessible during search and is evaluated by the trusted server only after the optimizer nominates a candidate.

“The test partition is inaccessible during search and is evaluated by the trusted server only after the optimizer nominates a candidate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All figures use the same run store and analysis set as the body.

“All figures use the same run store and analysis set as the body.”

Classified as probably not a substantial claim. May combine two claims.

Scores are means over K = 3 rounds, with ± denoting the standard error across round means, and timeouts count as zero.

“Scores are means over K = 3 rounds, with ± denoting the standard error across round means; timeouts count as zero.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The trusted gateway meters each evaluation's token usage as an input, cached, output, and total split, and stamps request-log records so that per-trial token attribution is reproducible from the run artifacts.

“The trusted gateway meters each evaluation's token usage as an input, cached, output, and total split, and stamps request-log records so that per-trial token attribution is reproducible from the run artifacts.”

Classified as probably not a substantial claim. May combine two claims.

Every candidate an optimizer produces is an immutable Git commit, dependencies are pinned by lockfile, and each evaluation runs in an isolated sandbox with per-case timeouts taken from the dataset's own declared agent clock.

“Every candidate an optimizer produces is an immutable Git commit, dependencies are pinned by lockfile, and each evaluation runs in an isolated sandbox with per-case timeouts taken from the dataset's own declared agent clock.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The seed harness is itself a task-specific prior.

“The seed harness is itself a task-specific prior.”

Classified as probably not a substantial claim.

Every ± reported in Table 3 is the standard error of that cell's three round means (sd/√K), i.e. re-run variation rather than case-to-case spread.

“Every ± is the standard error of that cell's three round means ( sd/ √ K ): re-run variation, not case-to-case spread.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The downstream corpora are public documents, and the paper introduces no personal or sensitive data.

“The downstream corpora are public documents, and we introduce no personal or sensitive data.”

Classified as probably not a substantial claim. May combine two claims.

In Table 3, an off-the-shelf column swaps a stock harness in for the seed program and changes nothing else (same dataset, same partition, same rounds, same target model), so the harness is the only variable.

“An off-the-shelf column swaps a stock harness in for the seed program and changes nothing else - same dataset, same partition, same rounds, same target model - so the harness is the only variable”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Candidates are restricted to Python and each task uses one pinned target model.

“candidates are restricted to Python and each task uses one pinned target model.”

May combine two claims.

Two further estimators of the model effect (gain standardized within task, and mean within-group rank) produce the identical ordering to the reported gain-unit estimator (pairwise rank concordance ≥ 1.00), so the ranking does not depend on the additive assumption.

“Two further estimators of the same quantity are computed and agree with it exactly on the ordering, so the ranking does not rest on the additive assumption; only the gain-unit estimator is reported”

• Also stated in: § C Model effects · p. 20 · plain

  “Two further estimators of the same quantity - gain standardized within task, and mean within-group rank - are computed but not shown; all three produce the identical ordering (pairwise rank concordance ≥ 1.00 ), so the ranking here does not depend on the additive assumption.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The parenthetical ranges reported in Table 2 are the range observed across a contestant's rounds, not confidence intervals, because two rounds do not estimate dispersion.

“A parenthetical is the range observed across that contestant's rounds, not a confidence interval - two rounds do not estimate dispersion.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The suite contains both regimes but does not vary seed complexity systematically.

“Our suite contains both regimes but does not vary seed complexity systematically.”

Classified as probably not a substantial claim.

The search budget is denominated in evaluation calls and case-runs so that comparisons do not implicitly favor optimizers with larger compute budgets.

“we keep the search budget denominated in evaluation calls and case-runs so that comparisons do not implicitly favor optimizers with larger compute budgets.”

May combine two claims.

The strongest models search broadly, yet their gains remain task-dependent and often too close to support a fine-grained ranking.

“The strongest search broadly, yet their gains remain task-dependent and often too close to support a fine-grained ranking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

HARNESSOPT-BENCH is designed to be hack-resistant, not hackproof.

“HARNESSOPT-BENCH is designed to be hack-resistant, not hackproof.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper reports the completed evaluation, including repeated runs, with explicit caveats and avoids over-claiming fine-grained model rankings.

“we report the completed evaluation, including repeated runs, with explicit caveats (see the Limitations section) and avoid over-claiming fine-grained model rankings.”

Probably not a substantial claim: may not pass S4 (based).

The score is the task's own verifier on a held-out split and nothing else.

“the score is the task's own verifier on a held-out split and nothing else”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Improving a mature agent tests diagnosis and refinement; starting from a stub tests construction.

“Improving a mature agent tests diagnosis and refinement; starting from a stub tests construction.”

Classified as probably not a substantial claim. May combine two claims.

The resolution column is the estimator's own measured split-round swing and the tier column cuts at it, so a shared tier means models are closer together than re-running the grid moves them, and ranks within a tier are not resolved.

“resolution is the estimator's own measured split-round swing and the tier column cuts at it, so a shared tier means 'closer together than re-running the grid moves them' and ranks within a tier are not resolved.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The optimizer cannot access the test partition or alter the target model, environment, or verifier.

“The optimizer cannot access the test partition or alter the target model, environment, or verifier”

May combine two claims.

Every optimizer runs inside a trusted harness that meters spend, versions every change for audit, and confines the target to a fixed model behind an allow-list.

“every optimizer runs inside a trusted harness that meters spend, versions every change for audit, and confines the target to a fixed model behind an allow-list”

May combine two claims.

Reading only the shared harnesses would miss the GAIA pattern and would suggest that sensitivity to the harness scales with model capability; adding each model's native harness removes that pattern.

“Reading only the shared harnesses would miss this, and would suggest that sensitivity to the harness scales with model capability; adding each model's native harness removes that pattern.”

May combine two claims.

The GAIA seed agent is a non-functional stub, so its baseline is a measured zero and gain on GAIA is the raw held-out score rather than a fraction of headroom.

“§ this seed is a non-functional stub, so its baseline is a measured zero and gain there is the raw held-out score - the one task on which an optimizer's result and a stock harness's score are directly comparable.”

• Also stated in: § D Supporting figures · p. 21 · plain

  “‡ GAIA starts from a non-functional stub.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In 7 entries the contestant shipped the unmodified seed on at least one round, so that round re-measures the seed rather than a failed edit.

“In 7 entries the contestant shipped the unmodified seed on at least one round, so that round re-measures the seed rather than a failed edit.”

May combine two claims.

Breadth is correlated with total modification volume, so the data do not isolate breadth from search effort.

“Breadth is also correlated with total modification volume, so the data do not isolate breadth from search effort.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the figures, GAIA gain is its raw held-out score and is not pooled with the other tasks.

“As in the body, GAIA gain is its raw held-out score and is not pooled with the other tasks.”

May combine two claims.

Current frontier models can diagnose, modify, and improve an agent as measured via a held-out reward, but unevenly.

“Current frontier models can, but unevenly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No harness effect is reported in the model-effect table because that scope holds one harness, so the fit has no harness factor to estimate.

“No harness effect is reported: this scope holds one harness, so the fit has no harness factor to estimate”

Classified as probably not a substantial claim. May combine two claims.

Each task pins its dataset by immutable reference and its split by a committed manifest.

“Each task pins its dataset by immutable reference and its split by a committed manifest.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The benchmark reuses existing public datasets and benchmarks under their respective licenses and adds seed agents and split definitions that the authors release.

“Our benchmark reuses existing public datasets and benchmarks under their respective licenses and adds seed agents and split definitions that we release.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Splits are exact 20/40/40 with no overlap, and the split generator verifies the committed tree byte-for-byte.

“Splits are exact 20/40/40 with no overlap, and the split generator verifies the committed tree byte-for-byte.”

Probably not a substantial claim: may not pass S2 (checkable).

Harness engineering is becoming a model capability, not merely infrastructure around one.

“Harness engineering is becoming a model capability, not merely infrastructure around one.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Repeated development and validation feedback may still reward strategies specific to a fixed evaluation.

“repeated development and validation feedback may still reward strategies specific to a fixed evaluation.”

May combine two claims.

The figure cannot distinguish selection-induced overfitting from a validation-test mismatch, so the paper claims only that the visible best score is optimistic.

“The figure cannot distinguish selection-induced overfitting from a validation-test mismatch, so we claim only that the visible best score is optimistic.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Baselines are the seed agent's held-out score pooled over K = 3 rounds and are re-pinned whenever a seed commit moves, using a script that reuses the original evaluation path.

“Baselines are the seed agent's held-out score pooled over K = 3 rounds and are re-pinned whenever a seed commit moves, using a script that reuses the original evaluation path.”

Classified as probably not a substantial claim.

The seed agents, split manifests, build configurations, and evaluation harness are released with the benchmark.

“The seed agents, split manifests, build configurations, and evaluation harness are released with the benchmark.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The off-the-shelf harness columns in Table 3 provide an indication of the seed harness' headroom.

“They provide an indication of the seed harness' headroom.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Every quantity reported about an optimizer's process is recomputed from its own execution trace and from the evaluator's record of which evaluations it ran, so the analysis is reproducible from the released artifacts without re-running any search.

“Every quantity we report about an optimizer's process is recomputed from its own execution trace and from the evaluator's record of which evaluations it ran, so the analysis is reproducible from the released artifacts without re-running any search.”