Evo-Bench: Can Language Models Improve Agent Harness?

Probably not a substantial claim: may not pass S4 (based).

GPT-5.6-Sol reacts superficially to aggregate scores rather than distilling causal failure modes from extensive logs.

“First, it reacts superficially to aggregate scores rather than distilling causal failure modes from extensive logs.”

May combine two claims.

Gains are highly domain-dependent, as evolvers effectively replicate web navigation in Search for massive gains and can even surpass manual engineering in General tasks, but struggle in Office tasks that demand highly specific processing workflows.

“gains are highly domain-dependent, as evolvers effectively replicate web navigation in Search for massive gains and can even surpass manual engineering in General tasks, but struggle in Office tasks that demand highly specific processing workflows”

May combine two claims.

Synthesized harnesses act as transferable reasoning structures, demonstrating cross-policy robustness by driving consistent gains across diverse policy models from Qwen, DeepSeek, and GLM.

“synthesized harnesses act as transferable reasoning structures, demonstrating cross-policy robustness by driving consistent gains across diverse policy models from Qwen, DeepSeek, and GLM.”

May combine two claims.

Top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines.

“top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines.”

May combine two claims.

GPT-5.6-Sol secures the highest score but dominates the cost spectrum by exceeding 500 USD per run.

“GPT-5.6-Solsecures the highest score but dominates the cost spectrum by exceeding 500 USD per run.”

May combine two claims.

Figure 4 reveals two distinct evolutionary behaviors: exhaustive exploration and early saturation.

“Figure 4 reveals two distinct evolutionary behaviors: exhaustive exploration and early saturation.”

May combine two claims.

For Office, GPT-5.6-Sol separates APEX and GDPval: APEX tracks evidence by file, page, and sheet cell and recomputes key values, whereas GDPval requires artifact creation, recalculation, reopening, and rendered inspection.

“For Office, it separates APEX and GDPval: APEX tracks evidence by file, page, and sheet cell and recomputes key values, whereas GDPval requires artifact creation, recalculation, reopening, and rendered inspection.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For Search, GPT-5.6-Sol implements web-search/fetch tools and a webpage cleaner that removes scripts, styles, and markup noise while preserving links.

“For Search, it implements web-search/fetch tools and a webpage cleaner that removes scripts, styles, and markup noise while preserving links.”

May combine two claims.

Autonomous evolution outperforms artificial harness in General tasks and excels in Search tasks, but struggles in Office tasks that demand highly specific processing workflows.

“while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows.”

May combine two claims.

Synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.

“the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.”

May combine two claims.

Search tasks are more amenable to optimization, with Claude Opus-4.8 gaining +34.8 to nearly match the Artificial harness, indicating evolvers easily synthesize missing web-navigation logic.

“Search tasks are more amenable to optimization, with Claude Opus-4.8 gaining +34.8 to nearly match the Artificial harness, indicating evolvers easily synthesize missing web-navigation logic.”

May combine two claims.

Top evolvers like GPT-5.6 Sol and Claude Opus 4.8 achieve substantial gains of 16.6 and 16.1 over the seed harness, respectively, closely approaching the human-engineered baseline of 47.5.

“top evolvers like GPT-5.6 Sol and Claude Opus 4.8 achieve substantial gains of 16.6 and 16.1 over the seed harness, respectively, closely approaching the human-engineered baseline of 47.5.”

May combine two claims.

GPT-5.6-Sol and Claude Opus-4.8 lead with scores of 46.3 and 45.8 respectively, yielding massive absolute gains over the initial CodeAct baseline.

“GPT-5.6-Sol and Claude Opus-4.8 lead with scores of 46.3 and 45.8 respectively, yielding massive absolute gains over the initial CodeAct baseline.”

May combine two claims.

GPT-5.6-Sol demonstrates tool construction, failure diagnosis, and iterative correction.

“It further demonstrates tool construction, failure diagnosis, and iterative correction.”

May combine two claims.

DeepSeek-V4-Pro anchors the absolute baseline, successfully synthesizing functional harness improvements for less than a single dollar.

“At the extreme budget end, DeepSeek-V4-Pro anchors the absolute baseline, successfully synthesizing functional harness improvements for less than a single dollar.”

May combine two claims.

The paper provides a systematic scientific account of harness evolution across 9 frontier models, characterizing substantial gains reaching 16.6 points, temporal anomalies like early saturation, and the robust transferability of evolved structures across distinct policy architectures.

“We provide a systematic scientific account of harness evolution across 9 frontier models, characterizing substantial gains reaching 16.6 points, temporal anomalies like early saturation, and the robust transferability of evolved structures across distinct policy architectures.”

May combine two claims.

GPT-5.6-Sol and Kimi-K2.7-Code uniquely exhaust the maximum 20-iteration budget with the highest step counts and longest durations.

“GPT-5.6-Sol and Kimi-K2.7-Codeuniquely exhaust the maximum 20-iteration budget with the highest step counts and longest durations.”

May combine two claims.

Despite its early halt, Qwen3.7-Max delivers a highly competitive score and dominates the General tasks domain, demonstrating exceptional sample efficiency in synthesizing effective updates without exhaustive trial and error.

“Despite its early halt, Qwen3.7-Maxdelivers a highly competitive score and dominates the General tasks domain, demonstrating exceptional sample efficiency in synthesizing effective updates without exhaustive trial and error.”

May combine two claims.

GPT-5.6-Sol underutilizes the research budget, lacking the goal-oriented persistence of human engineers.

“Third, it underutilizes the research budget, lacking the goaloriented persistence of human engineers.”

May combine two claims.

Office tasks remain stubborn; most models show marginal improvements or slight regressions, failing to match the Artificial baseline likely due to the difficulty of discovering specialized workflows.

“Conversely, Office tasks remain stubborn.Most models show marginal improvements or slight regressions, failing to match the Artificial baseline likely due to the difficulty of discovering specialized workflows.”

May combine two claims.

GLM-5.2 demonstrates substantial diagnosis and tool engineering but relies more on domain-specific guidance and local controls, while Qwen3.7-Max and MiniMax M3 focus primarily on web tools, prompt specialization, and safety filters.

“GLM-5.2 demonstrates substantial diagnosis and tool engineering but relies more on domain-specific guidance and local controls, while Qwen3.7-Maxand MiniMax M3 focus primarily on web tools, prompt specialization, and safety filters.”

Borderline on S3 (load-bearing), S4 (based): unclear from the text.

The evolutionary gains are highly uneven across different task domains.

“The evolutionary gains are highly uneven across different task domains.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CodeAct baseline achieves an Overall score of 29.7.

“- | CodeAct | 11.7 | 38.4 | 48.4 | 29.7 | -”

Probably not a substantial claim: may not pass S3 (load-bearing).

The four evolutions yield 12 auxiliary harnesses.

“The four evolutions therefore yield 12 harnesses, H aux , completing the auxiliary-harness selection step of Stage 1.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

APEX pool n = 421; selected n = 96; med.Sens = 0.38; med.Perf = 0.

“pool n = 421; selected n = 96 med.Sens = 0.38; med.Perf = 0”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BrowseComp pool n = 768; selected n = 160; med.Sens = 0.22; med.Perf = 0.22.

“BrowseComp pool n = 768; selected n = 160 med.Sens = 0.22; med.Perf = 0.22”

Probably not a substantial claim: may not pass S3 (load-bearing).

Content hashing reduces the 73 evaluated harnesses to 65 unique revisions.

“Content hashing reduces the 73 evaluated harnesses to 65 unique revisions; the common seed harness is excluded.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The headroom-horizon selection selects 128 Search, 128 Office, and 64 General tasks.

“This selects 128 Search, 128 Office, and 64 General tasks, favoring headroom and long interaction horizons (Figure 10).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

DeepSeek-V4-Pro's best validation score was 46.5 at iteration 3 and its frozen score was 42.6 at iteration 15.

“DeepSeek-V4-Pro | 15/20 | 202/1000 | 4 | 46.5 (I3) → 42.6 (I15)”

May combine two claims.

Expanding the budget from 24 to 48 hours yields consistent, monotonic improvements in both the Overall Score and the Anytime Validation score.

“Expanding the budget from 24 to 48 hours yields consistent, monotonic improvements in both the Overall Score and the Anytime Validation score.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GDPval pool n = 215; selected n = 96; med.Sens = 0.31; med.Perf = 0.

“GDPval pool n = 215; selected n = 96 med.Sens = 0.31; med.Perf = 0”

May combine two claims.

Both Qwen3.7-Max and GLM-5.2 demonstrate a clear positive scaling trend with expanding budget.

“As illustrated in Figure 7, both models demonstrate a clear positive scaling trend.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

HLE pool n = 768; selected n = 160; med.Sens = 0.29; med.Perf = 0.22.

“HLE pool n = 768; selected n = 160 med.Sens = 0.29; med.Perf = 0.22”

May combine two claims.

When utilizing Qwen3.6-35B-A3B as the policy model, the initial Overall score starts at 13.9, and both evolvers optimize the harness to reach scores of 27.9 and 29.2.

“For instance, when utilizing Qwen3.6-35B-A3B as the policy model, the initial Overall score starts at a modest 13.9.Yet, both evolvers successfully optimize the harness to reach scores of 27.9 and 29.2.”

May combine two claims.

Claude Opus 4.8 achieves an Overall score of 45.8, an improvement of +16.1 over the CodeAct baseline.

“2 | Claude Opus 4.8 Anthropic (2026b) | 46.5 +34.8 | 39.7 +1.3 | 56.3 +7.9 | 45.8 +16.1 | 51.4”

• cites Anthropic (2026b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Selection favors non-negative sensitivity while preserving difficulty coverage, and the two splits remain aligned.

“Selection favors non-negative sensitivity while preserving difficulty coverage, and the two splits remain aligned.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Artificial Harness achieves an Overall score of 47.5.

“- | Artificial Harness | 46.7 | 43.9 | 56.3 | 47.5 | -”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen3.6-27B's best validation score was 49.7 at iteration 10 and its frozen score was 45.4 at iteration 18.

“Qwen3.6-27B | 18/20 | 349/1000 | 16 | 49.7 (I10) → 45.4 (I18)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

After validity, rollout-coverage, deduplication, and cross-suite leakage checks, 11,322 candidates remain: 2,988 Search, 7,564 Office, and 770 General.

“After validity, rollout-coverage, deduplication, and cross-suite leakage checks, 11,322 candidates remain: 2,988 Search, 7,564 Office, and 770 General.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claw-Eval pool n = 157; selected n = 96; med.Sens = 0.29; med.Perf = 0.83.

“pool n = 157; selected n = 96 med.Sens = 0.29; med.Perf = 0.83”

May combine two claims.

A malformed I15 verifier crashed 159/160 tasks and reduced the score to 0.3.

“a malformed I15 verifier crashed 159/160 tasks and reduced the score to 0.3, a defect missed by local preflight.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Eight slightly negative Claw-Eval tasks are quota backfills from sparse strata.

“Eight slightly negative Claw-Eval tasks are quota backfills from sparse strata.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Kimi-K2.7-Code's best and frozen validation score was 45.9 at iteration 13.

“Kimi-K2.7-Code | 20/20 | 549/1000 | 52 | 45.9 (I13) → 45.9 (I13)”

May combine two claims.

GPT-5.6 Sol achieves an Overall score of 46.3, an improvement of +16.6 over the CodeAct baseline.

“1 | GPT-5.6 Sol OpenAI (2026) | 44.5 +32.8 | 41.6 +3.2 | 59.4 +11.0 | 46.3 +16.6 | 50.1”

• cites OpenAI (2026)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

BrowseComp, HLE, and APEX-Agents skew hard; GDPval is broader and Claw-Eval easier.

“BrowseComp, HLE, and APEX-Agents skew hard; GDPval is broader and Claw-Eval easier.”

Classified as probably not a substantial claim.

The custom evolve harness is implemented natively within Evo-Bench to enable a controlled and transparent evaluation protocol.

“We implement this custom harness natively within Evo-Bench to enable a controlled and transparent evaluation protocol.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Harness evolution is generally considered the first step toward achieving self-improvement.

“Among these lines of research, harness evolution is generally considered the first step toward achieving self-improvement Weng (2026).”

• cites Weng (2026)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The quote only partly states this claim.

Autonomously evolved reasoning structures can outstrip manually crafted solutions.

“This milestone demonstrates that autonomously evolved reasoning structures can indeed outstrip manually crafted solutions.”

May combine two claims.

Evo-Bench employs a harness-guided construction framework that leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization.

“Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization.”

Classified as probably not a substantial claim.

Evo-Bench is a benchmark for evaluating large language models' harness-evolving capability: the ability to conduct long-horizon, code-centric iterative improvement of an executable agent harness.

“Evo-Bench is a benchmark for evaluating large language models' harness-evolving capability: the ability to conduct long-horizon, code-centric iterative improvement of an executable agent harness.”

Probably not a substantial claim: may not pass S4 (based).

Tasks with non-positive Sens(x) provide limited signal for distinguishing harness quality.

“We first remove tasks with non-positive Sens(x), as they provide limited signal for distinguishing harness quality.”

May combine two claims.

The evolved harness relies on localized modifications, leaving core components primitive: the planner acts passively, the context remains append-only, and the verifier stays entirely permissive.

“Consequently, the evolved harness relies on localized modifications, leaving core components primitive: the planner acts passively, the context remains append-only, and the verifier stays entirely permissive.”

May combine two claims.

Evo-Bench shifts the evaluation paradigm by explicitly benchmarking foundation models as long-horizon research agents tasked with evolving a single, shared general-purpose harness, validated under a rigorous, sensitivity-calibrated task split.

“Instead of evaluating task-specific artifacts or unconstrained self-evolution, we explicitly benchmark foundation models as long-horizon research agents tasked with evolving a single, shared general-purpose harness, validated under a rigorous, sensitivity-calibrated task split.”

May combine two claims.

The early peak suggests that Claude Opus-4.8 and GLM-5.2 rapidly evolve high-quality structures but frequently introduce detrimental modifications in subsequent iterations.

“This early peak suggests they rapidly evolve high-quality structures but frequently introduce detrimental modifications in subsequent iterations.”

Probably not a substantial claim: may not pass S4 (based).

Randomly splitting tasks within each stratum into validation and evaluation suites ensures that they follow the same difficulty distribution.

“Then we randomly split tasks within each stratum into validation and evaluation suites, ensuring that they follow the same difficulty distribution.”

Classified as probably not a substantial claim.

Higher Sens(x) indicates that task x consistently reflects the quality ranking of different harnesses, where stronger harnesses achieve better performance on the task.

“Higher Sens(x) indicates that task x consistently reflects the quality ranking of different harnesses, where stronger harnesses achieve better performance on the task.”

May combine two claims.

High variance across harnesses only captures the magnitude of performance differences and does not indicate whether these differences reflect the relative quality of harnesses.

“However, high variance only captures the magnitude of performance differences and does not indicate whether these differences reflect the relative quality of harnesses.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Selecting tasks with lower average scores and longer interaction trajectories indicates greater room for harness improvements and stronger signals for harness evolution.

“select tasks with lower average scores and longer interaction trajectories, which indicate greater room for harness improvements and stronger signals for harness evolution.”

Probably not a substantial claim: may not pass S4 (based).

To ensure that validation-side optimization reliably predicts held-out evaluation performance, benchmark tasks should satisfy two key properties: Harness Sensitivity and Cross-Suite Alignment.

“To ensure that validation-side optimization reliably predicts held-out evaluation performance, benchmark tasks should satisfy two key properties: Harness Sensitivity, where task performance is sensitive to harness improvements while controlling for the underlying model, and Cross-Suite Alignment, where validation and evaluation suites exhibit consistent performance to harness variation.”

Classified as probably not a substantial claim.

The evolve harness operates as a fixed agent loop explicitly designed to support long-horizon autonomous work.

“The evolve harness operates as a fixed agent loop explicitly designed to support long-horizon autonomous work.”

May combine two claims.

The task suite construction procedure yields a task set that preserves difficulty diversity while prioritizing tasks that are more responsive to harness evolution.

“This procedure yields a task set that preserves difficulty diversity while prioritizing tasks that are more responsive to harness evolution.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The dataset block gives task families and 2:2:1 aggregation, but no held-out instances.

“The dataset block gives task families and 2:2:1 aggregation, but no held-out instances.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Deterministic scorers use no prompt; model judges use Section C.1 and receive references only inside evaluation.

“Deterministic scorers use no prompt; model judges use Section C.1 and receive references only inside evaluation (Table 7).”

Classified as probably not a substantial claim.

The fixed evolve harness H evo lets every evolver inspect, revise, and evaluate a policy harness under the same prompt, tools, state management, and resource accounting.

“The fixed evolve harness H evo lets every evolver inspect, revise, and evaluate a policy harness under the same prompt, tools, state management, and resource accounting.”

Classified as probably not a substantial claim.

We apply the same Codex-based semantic audit to the complete evolver and policy trajectories of every evolution run.

“Because pattern matching alone cannot reliably detect such behavior, we apply the same Codex-based semantic audit to the complete evolver and policy trajectories of every evolution run.”

• AssumptionRests on an assumption: Pattern matching alone cannot reliably detect detector-evasion behavior.

Classified as probably not a substantial claim. May combine two claims.

The audit checks whether protected information reached policy execution, whether retrieval targeted benchmark-specific artifacts, and whether behavior attempted to conceal detector-visible evidence.

“The audit checks whether protected information reached policy execution, whether retrieval targeted benchmark-specific artifacts, and whether behavior attempted to conceal detector-visible evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Selecting the same number of harnesses from each evolution prevents evolutions with more evaluated revisions from dominating the selected set.

“Selecting the same number of harnesses from each evolution prevents evolutions with more evaluated revisions from dominating the selected set.”

Probably not a substantial claim: may not pass S4 (based).

Live state comes from tools rather than self-report.

“Live state comes from tools rather than self-report.”

Probably not a substantial claim: may not pass S4 (based).

By establishing a rigorous framework anchored in controlled attribution, long-horizon iteration, and transfer alignment, Evo-Bench disentangles genuine harness-evolving capability from trivial prompt tuning and stochastic runtime noise.

“By establishing a rigorous framework anchored in controlled attribution, long-horizon iteration, and transfer alignment, Evo-Bench disentangles genuine harness-evolving capability from trivial prompt tuning and stochastic runtime noise.”

Classified as probably not a substantial claim. May combine two claims.

The evolver may read validation artifacts but can write only to its policy harness and workbench.

“The evolver may read validation artifacts but can write only to its policy harness and workbench.”

Classified as probably not a substantial claim. May combine two claims.

Each policy rollout receives an immutable harness snapshot, a public task view with answers and scorer fields removed, and a fresh workspace; validation artifacts, evolver notes, sibling workspaces, and held-out data are not mounted.

“Each policy rollout instead receives an immutable harness snapshot, a public task view with answers and scorer fields removed, and a fresh workspace; validation artifacts, evolver notes, sibling workspaces, and held-out data are not mounted.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Judge instructions are fixed and post-rollout.

“Judge instructions are fixed and post-rollout.”

Classified as probably not a substantial claim.

Scorers, aggregation logic, suite definitions, and iteration accounting remain in the fixed host-side framework.

“Scorers, aggregation logic, suite definitions, and iteration accounting remain in the fixed host-side framework.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The auxiliary task selection process is independent of the CodeAct seed.

“This process is independent of the CodeAct seed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The check applies to information delivered to the policy rather than to attempted queries: an unsuccessful request is not penalized, and a request blocked before protected content reaches the policy is not treated as a violation.

“The check applies to information delivered to the policy rather than to attempted queries: an unsuccessful request is not penalized, and a request blocked before protected content reaches the policy is not treated as a violation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Evolved prompt and code are frozen together.

“Evolved prompt and code are frozen together.”

Classified as probably not a substantial claim. May combine two claims.

Every formal validation evaluation consumes one iteration, and held-out evaluation starts only after the selected harness is frozen and returns no evidence to the evolver.

“Every formal validation evaluation consumes one iteration, and held-out evaluation starts only after the selected harness is frozen and returns no evidence to the evolver.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For a confirmed match, we record the source and tool event, mark the trial as a policy violation, and set all native score fields to zero before computing domain scores and Overall Score.

“For a confirmed match, we record the source and tool event, mark the trial as a policy violation, and set all native score fields to zero before computing domain scores and Overall Score.”

Classified as probably not a substantial claim.

Evo-Bench is a novel benchmark designed to evaluate language models' intrinsic capacity to autonomously improve agent harnesses.

“In this work, we present Evo-Bench, a novel benchmark designed to evaluate language models' intrinsic capacity to autonomously improve agent harnesses.”

May combine two claims.

Allocating greater computational resources for iterative exploration can still provide meaningful performance dividends, ultimately enabling the synthesis of more refined agent harnesses.

“This positive scaling trend indicates that despite the previously noted limitations in budget utilization, allocating greater computational resources for iterative exploration can still provide meaningful performance dividends, ultimately enabling the synthesis of more refined agent harnesses.”

Classified as probably not a substantial claim.

Policy and judge are fixed.

“Policy and judge are fixed.”