VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Validation accuracy can also guide hyperparameter and architecture search; both uses reduce the validation set to a single number.

“Validation accuracy can also guide hyperparameter and architecture search (Snoek et al., 2012;Zoph & Le, 2017); both uses reduce the validation set to a single number.”

• cites Snoek et al., 2012 sourceSupported
• cites Zoph & Le, 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The four methods differ in the evidence interface through which the optimizer digests failures.

“Evidence interfaces.The four methods differ in the evidence interface through which the optimizer digests failures: Meta-Harness reads raw traces directly, AHE distills them into per-task root-cause reports, Self-Harness mines model-specific weakness patterns, and HarnessX summarizes trajectories on demand.”

• cites Lee et al., 2026bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lin et al., 2026 sourcePartly supported

  Section 2.2 says automated agent optimizers differ in the evidence they observe, and Section 3.2 describes AHE’s per-task root-cause reports. The cited paper does not establish the stated evidence interfaces for Meta-Harness, Self-Harness, or HarnessX, so it supports only part of the four-method comparison. No page numbers are available in our copy. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Zhang et al., 2026aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The original papers use different evaluation protocols.

“Evaluation protocols of the original papers.The original papers use different evaluation protocols.”

• cites Lee et al., 2026bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lin et al., 2026 sourceCould not verify

  Section 4.1 Setup and Appendix A describe this paper’s evaluation protocol, and §4.3 says its cross-benchmark baselines were evaluated under identical infrastructure. No passage found states that the original papers used different evaluation protocols (searched for “original papers,” “protocols,” and differing baseline evaluation settings). (LLM, conf=0.69)

  Experimental — accuracy not yet measured
• cites Chen et al., 2026cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Learning g treats the optimizer as a learned algorithm, using execution feedback for program revision.

“Learning g treats the optimizer as a learned algorithm (Zakerinia et al., 2024), using execution feedback for program revision (Agrawal et al., 2026).”

• cites Zakerinia et al., 2024 sourceSupported
• cites Agrawal et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available.

“In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available.”

• Also stated in: § 2.3 TWO OBSERVATIONS ON SELF-EVOLVING OPTIMIZERS · hedged

  “With the verification tool, the same optimizer could run a draft on a training task before submission and reached 41.05% test accuracy, the highest in Table 1.”

May combine two claims.

When self-evolution was added to Meta-Harness, the optimizer wrote untested lessons into its own harness, and no later round beat the initial harness on validation.

“When we added self-evolution to Meta-Harness (Lee et al., 2026b), the optimizer wrote untested lessons into its own harness, and no later round beat the initial harness on validation.”

• Also stated in: § 2.3 TWO OBSERVATIONS ON SELF-EVOLVING OPTIMIZERS · plain

  “Without verification, adding self-evolution to Meta-Harness (Lee et al., 2026b) made it worse: no round beat round 0 on validation, so validation selected the initial harness over every evolved one (r ⋆ = 0 in Table 1).”

• cites Lee et al., 2026bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

VERSE's best validation-selected harness reaches 42.3% and 37.7% accuracy on heldout SWE-rebench and OOD tasks, respectively, against 39.2% and 29.3% for the strongest baselines.

“Its best validation-selected harness reaches 42.3% and 37.7% accuracy, respectively, against 39.2% and 29.3% for the strongest baselines.”

May combine two claims.

VERSE's selected and final harnesses stay within 1.6 points in distribution, but differ by 5.0 points on HarnessX out of distribution.

“Its selected and final harnesses stay within 1.6 points in distribution, but differ by 5.0 points on HarnessX out of distribution.”

May combine two claims.

The VERSE tools already help without self-evolution: in distribution, they select a better harness than the baseline on three of the four hosts, by 4.0 to 10.5 points.

“The VERSE tools already help without self-evolution: in distribution, they select a better harness than the baseline on three of the four hosts, by 4.0 to 10.5 points.”

May combine two claims.

Few draft edits fix their targets, and execution shows it before submission.

“Few draft edits fix their targets, and execution shows it before submission.”

May combine two claims.

Removing the verification tools barely changes the selected-round accuracy: on this host, the fixed optimizer's first nine calls all refuted its drafts, and it stopped calling after the second round.

“Removing them barely changes the selected-round accuracy: on this host, the fixed optimizer's first nine calls all refuted its drafts, and it stopped calling after the second round.”

May combine two claims.

With self-evolution, VERSE scores above its unequipped baseline in all 16 reported comparisons of Table 2, on both test sets and for both the selected and the final harness, by 0.6 to 10.3 points.

“With self-evolution, VERSE scores above its unequipped baseline in all 16 reported comparisons of Table 2, on both test sets and for both the selected and the final harness, by 0.6 to 10.3 points.”

May combine two claims.

Across the four hosts of Table 2, self-evolving optimizers call the tools more than twice as often, and more than three times as often in the last three rounds.

“Across the four hosts of Table 2, self-evolving optimizers call the tools more than twice as often, and more than three times as often in the last three rounds.”

May combine two claims.

On the in-distribution test sets of Tables 123, the validation-selected harness matches or beats the final-round harness in 15 of 18 runs, by up to 6.8 points, and trails it by at most 1.6 points.

“On the in-distribution test sets of Tables 123, the validation-selected harness matches or beats the final-round harness in 15 of 18 runs, by up to 6.8 points, and trails it by at most 1.6 points.”

May combine two claims.

Self-evolution does not raise every score: compared with VERSE without self-evolution, it raises the selected accuracy on Meta-Harness on both test sets and on HarnessX out of distribution, and lowers it in the other five cases.

“Self-evolution does not raise every score: compared with VERSE without self-evolution, it raises the selected accuracy on Meta-Harness on both test sets and on HarnessX out of distribution, and lowers it in the other five cases.”

May combine two claims.

With each submitted edit, the optimizer names the failing training tasks it expects the edit to fix; only one in ten of these tasks passed in the next training run.

“With each submitted edit, the optimizer names the failing training tasks it expects the edit to fix; only one in ten of these tasks passed in the next training run.”

May combine two claims.

In Appendix H, random or fixed experiment schedules and a restricted workspace also cost accuracy.

“In Appendix H, random or fixed experiment schedules and a restricted workspace also cost accuracy.”

May combine two claims.

Evaluated drafts fixed a failing target at similar rates in both groups (9 of 81 against 8 of 89); the clearer difference lies in what the optimizer tests and how it responds when a test fails.

“Evaluated drafts fixed a failing target at similar rates in both groups (9 of 81 against 8 of 89); the clearer difference lies in what the optimizer tests and how it responds when a test fails.”

May combine two claims.

Without self-evolution, the VERSE tools fall below the baseline in 8 of the 16 comparisons of Table 2, while VERSE with self-evolution stays above it in all 16.

“Without self-evolution, however, they fall below the baseline in 8 of the 16 comparisons of Table 2, while VERSE with self-evolution stays above it in all 16.”

May combine two claims.

On the OOD test set, the validation-selected baselines stay within two points of blank, including Meta-Harness, whose gain over blank falls from 5.2 points in distribution to 1.2.

“On the OOD test set, the validation-selected baselines stay within two points of blank, including Meta-Harness, whose gain over blank falls from 5.2 points in distribution to 1.2.”

May combine two claims.

Without the reports, the optimizer calls the verification tools more than three times as often, yet selects a weaker harness.

“Without the reports, the optimizer calls the verification tools more than three times as often, yet selects a weaker harness.”

May combine two claims.

What VERSE learns on Python tasks carries over to Go, Java, Rust, and TypeScript: across the five languages, it raises its host in 16 of the 20 pairs of host and language.

“What VERSE learns on Python tasks thus carries over to Go, Java, Rust, and TypeScript: across the five languages, it raises its host in 16 of the 20 pairs of host and language (Appendix G.1).”

May combine two claims.

In Table 3, removing attribution and minimization, which also removes the training audit, costs about half of the gain that VERSE adds to AHE (2.2 of 4.0 points); removing the training audit alone costs 1.2.

“In Table 3, removing attribution and minimization, which also removes the training audit, costs about half of the gain that VERSE adds to AHE (2.2 of 4.0 points); removing the training audit alone costs 1.2.”

May combine two claims.

Self-evolving optimizers also respond when a check fails: when the tool showed that a draft fixed none of its targets, they revised the draft before submitting in 30 of 38 cases (79%), against 42 of 74 (57%) without self-evolution.

“Self-evolving optimizers also respond when a check fails: when the tool showed that a draft fixed none of its targets, they revised the draft before submitting in 30 of 38 cases (79%), against 42 of 74 (57%) without self-evolution.”

May combine two claims.

In the evolution logs, few draft edits fix their targets, and self-evolving optimizers add regression checks.

“In the evolution logs, few draft edits fix their targets, and self-evolving optimizers add regression checks (Section 4).”

May combine two claims.

The self-evolving optimizer on the AHE host ends each candidate after a median of 26 turns instead of 58, so it makes fewer optimizer calls than the baseline despite its self-updates.

“The self-evolving optimizer on the AHE host ends each candidate after a median of 26 turns instead of 58, so it makes fewer optimizer calls than the baseline despite its self-updates.”

May combine two claims.

In the controlled study, self-evolution lowered accuracy without verification and gave the best result with it.

“In our controlled study, self-evolution lowered accuracy without verification and gave the best result with it (Section 2.3).”

May combine two claims.

The verification tools add 27 to 104 agent runs, 2 to 7% of a run's agent runs, and at most 37 replays.

“The verification tools add 27 to 104 agent runs, 2 to 7% of a run's agent runs, and at most 37 replays.”

May combine two claims.

The OOD test set of Table 2 is the July 2026 release of the SWErebench leaderboard, covering tasks from May 15 to July 1, 2026. It holds 111 tasks in Go (21), Java (21), Python (20), Rust (25), and TypeScript (24). Four Java tasks fail their reference solution in the environment and are excluded, so OOD accuracies use 107 tasks: Go 21, Java 17, Python 20, Rust 25, and TypeScript 24.

“The OOD test set of Table 2 is the July 2026 release of the SWErebench leaderboard, which covers tasks from May 15 to July 1, 2026.It holds 111 tasks in Go (21), Java (21), Python (20), Rust (25), and TypeScript (24).Four Java tasks fail their reference solution in our environment and are excluded, so OOD accuracies use 107 tasks: Go 21, Java 17, Python 20, Rust 25, and TypeScript 24.”

May combine two claims.

On the 87 tasks outside Python, blank scores 25.3%, and VERSE with self-evolution scores 35.6% on the Meta-Harness host and 37.9% on the HarnessX host, the two hosts with the largest OOD gains in Table 2.

“On the 87 tasks outside Python, blank scores 25.3%, and VERSE with self-evolution scores 35.6% on the Meta-Harness host and 37.9% on the HarnessX host, the two hosts with the largest OOD gains in Table 2.”

May combine two claims.

Of the 170 draft evaluations, 17 fixed a failing task, 30 passed only on already-passing tasks added as regression checks, 112 passed none of their tasks, and 11 were inconclusive.

“• in 17, the draft fixed a task that was failing; • in 30, it passed only on tasks that already passed, added by the optimizer as regression checks; • in 112, it passed none of its tasks; • 11 were inconclusive: 10 inconclusive verdicts and one pass whose task is missing from the log.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the four self-evolving runs of Table 2, simple verification was called 21 times, against 140 calls to the three verification tools.

“In the four self-evolving runs of Table 2, simple verification was called 21 times, against 140 calls to the three verification tools.”

May combine two claims.

The sealed test pool holds 110 tasks. Two of them fail their own official reference solution under the grading pipeline and are excluded, so in-distribution SWE-rebench accuracies use 108 tasks.

“The sealed test pool holds 110 tasks.Two of them fail their own official reference solution under our grading pipeline and are excluded, so in-distribution SWE-rebench accuracies use 108 tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the evolution runs, the verification tool made 170 draft evaluations.

“In the evolution runs of Tables 123, the verification tool made 170 draft evaluations.”

May combine two claims.

Removing only the training audit, with attribution and minimization still running, costs more, and its final round is the lowest in the table.

“Removing only the training audit, with attribution and minimization still running, costs more, and its final round is the lowest in the table.”

May combine two claims. The quote only partly states this claim.

The final harness with simple verification scored 41.05%, the highest in Table 1.

“the final harness scored 41.05%, the highest in Table 1”

May combine two claims.

The equipped version keeps its advantage over its baseline in three of four pairs on each executor, and transfer is larger for the executor that handles basic tool use worse under blank (Qwen).

“Two patterns remain.The equipped version keeps its advantage over its baseline in three of four pairs on each executor, and transfer is larger for the executor that handles basic tool use worse under blank (Qwen).”

May combine two claims.

Two faults injected at once flip 81.0% of runs, more than either fault alone.

“Two faults injected at once flip 81.0% of runs, more than either fault alone.”

May combine two claims.

At the same probe budget, a random schedule costs accuracy, and a fixed hand-written schedule falls below plain AHE at the selected round.

“At the same probe budget, a random schedule costs accuracy, and a fixed hand-written schedule falls below plain AHE at the selected round: probes the optimizer did not ask for can be worse than none.”

May combine two claims.

Restricting the self-evolving optimizer to only its prompts, only its notes and skills, or only its tools lowers selected-round accuracy by about three points in each case.

“Restricting the selfevolving optimizer to only its prompts, only its notes and skills, or only its tools lowers selectedround accuracy by about three points in each case.”

May combine two claims.

On SWE-rebench, adding the verification tools improves all four baselines at the selected round, and every equipped baseline reaches or exceeds the strongest unequipped one.

“On SWE-rebench, adding them improves all four baselines at the selected round, and every equipped baseline reaches or exceeds the strongest unequipped one.”

May combine two claims.

VERSE's three-run mean is the highest in both views on both benchmarks, and at the selected round each of its runs scores above the mean of every baseline.

“Its three-run mean is the highest in both views on both benchmarks, and at the selected round each of its runs scores above the mean of every baseline.”

The quote only partly states this claim.

Overall, 35.7% of 3,868 valid single-step fault injections flip the run to failure.

“Overall, 35.7% flip the run to failure”

May combine two claims.

With verification tools, the two views usually stay within two points of each other, although some runs lose more.

“With verification tools, the two views usually stay within two points of each other, although some runs lose more.”

May combine two claims.

Self-evolution matters most on the Meta-Harness and HarnessX hosts: without it, the evolved harnesses stay near their baselines outside Python; with it, they reach 35.6% and 37.9%.

“Self-evolution matters most on these two hosts: without it, the evolved harnesses stay near their baselines outside Python (28.7% and 26.8%, against 29.5% and 26.1%); with it, they reach 35.6% and 37.9%.”

May combine two claims.

With a budget of one or two units per candidate, most of the gain disappears; from four units on, most of it returns.

“With a budget of one or two units per candidate, most of the gain disappears; from four units on, most of it returns (Figure 8).”

May combine two claims.

In round 1, the optimizer classified the 79 failed training runs: 41 ran tests with a Python interpreter that lacked the test framework, 28 tried to install packages offline, and 26 tried to reach the network.

“In round 1, it classified the 79 failed training runs: 41 ran tests with a Python interpreter that lacked the test framework, 28 tried to install packages offline, and 26 tried to reach the network.”

May combine two claims.

VERSE raises its host in 16 of the 20 pairs of host and language, and in all five languages on the Meta-Harness and HarnessX hosts.

“VERSE raises its host in 16 of the 20 pairs of host and language, and in all five languages on the Meta-Harness and HarnessX hosts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The optimizer's notes, skills, and code were empty at the start.

“Its notes, skills, and code were empty at the start;”

May combine two claims.

Self-evolution alone gives no gain: its validation-selected harness scores the same as Meta-Harness (37.08%).

“Self-evolution alone gives no gain: its validation-selected harness scores the same as Meta-Harness (37.08%)”

May combine two claims.

Reading a trace identifies the kind of failure far more reliably than the failing step, so localization needs execution.

“The measurements below support these premises, and a fourth finding shapes the design: reading a trace identifies the kind of failure far more reliably than the failing step, so localization needs execution.”

Classified as probably not a substantial claim. May combine two claims.

The notes list this as the next lead, and the round-2 edit made the executor check its patch against the requirements stated in the issue and run the wider test directory before submitting.

“The notes list this as the next lead (Appendix L.4), and the round-2 edit made the executor check its patch against the requirements stated in the issue and run the wider test directory before submitting.”

May combine two claims.

Zero-shot, most transplanted harnesses stay near the blank, and adaptation recovers the blank level but rarely exceeds it.

“Zero-shot, most transplanted harnesses stay near the blank, and adaptation recovers the blank level but rarely exceeds it.”

May combine two claims.

Averaged over the four hosts, VERSE raises accuracy in every language: by 10.3 points in Go, 10.3 in Java, 2.3 in Rust, 4.9 in TypeScript, and 4.2 in Python.

“Averaged over the four hosts, it raises accuracy in every language: by 10.3 points in Go, 10.3 in Java, 2.3 in Rust, 4.9 in TypeScript, and 4.2 in Python.”

May combine two claims.

On Qwen, VERSE is the only method above the best baseline in all three columns, and its cold-started row also exceeds the best baseline in both adaptation rounds.

“The one consistent signal again favors the inherited optimizer harness: on Qwen, VERSE is the only method above the best baseline in all three columns, and its cold-started row also exceeds the best baseline in both adaptation rounds.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Terminal-Bench is less sensitive to the harness in this setting: most methods stay within about two points of blank.

“Terminal-Bench is less sensitive to the harness in this setting: most methods stay within about two points of blank, in line with Wang et al. (2026b).”

• cites Wang et al. (2026b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Outside Python, the four baselines range from 1.1 points below blank to 4.2 points above it, while VERSE with self-evolution is 3.4 to 12.6 points above blank.

“Outside Python, the four baselines range from 1.1 points below blank to 4.2 points above it, while VERSE with self-evolution is 3.4 to 12.6 points above blank.”

May combine two claims.

In round 2, all three verification calls included a passing task, compared with none of the 3 calls in round 1 and none of the 12 calls in the run with the same host and tools but without self-evolution.

“In round 2, all three verification calls included a passing task, compared with none of the 3 calls in round 1 and none of the 12 calls in the run with the same host and tools but without self-evolution.”

May combine two claims.

When the new harness ran on the training tasks, the three counts fell to 2, 0, and 0, and training accuracy rose from 31 to 37 of 110 tasks.

“When the new harness ran on the training tasks, the three counts fell to 2, 0, and 0, and training accuracy rose from 31 to 37 of 110 tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Validation selected round 2.

“Validation selected round 2.”

May combine two claims.

The round-1 edit fixed three validation tasks but broke two that had passed.

“The round-1 edit fixed three validation tasks but broke two that had passed.”

• Also stated in: § Additional information · plain

  “The accepted change-set's transforming hooks fixed 3 tasks but REGRESSED 2 previously-passing ones”

May combine two claims.

Adaptation helps most when the starting harness is weak on the new executor and little when it is already strong.

“Adaptation helps most when the starting harness is weak on the new executor and little when it is already strong, as for HarnessX + VERSE on Qwen.”

May combine two claims.

Under blank, Qwen never uses an editor tool and writes every patch through shell heredocs.

“Under blank, Qwen never uses an editor tool and writes every patch through shell heredocs; in all 8 tasks that the transplant flips, the blank trajectory dies on heredoc quoting before the patch ever lands.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The reverse direction moves less, consistent with how flat Terminal-Bench is.

“The reverse direction moves less, consistent with how flat Terminal-Bench is.”

May combine two claims.

On SWE-rebench, three of the four baselines lose accuracy from the selected to the final round, and a single run can lose several points.

“On SWE-rebench, three of the four baselines lose accuracy from the selected to the final round, and a single run can lose several points.”

May combine two claims.

The clear exception is the HarnessX family on Qwen: both variants sit above blank, and HarnessX + VERSE gives the largest zero-shot gain in the table.

“The clear exception is the HarnessX family on Qwen: both variants sit above blank, and HarnessX + VERSE gives the largest zero-shot gain in the table.”

May combine two claims.

VERSE is the strongest configuration on both benchmarks.

“VERSE is the strongest configuration on both benchmarks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The optimizer's count of about 110 shell calls per session is its own: round 1 was interrupted and restarted, and the log it read held both attempts, each with 53 to 62 shell calls per candidate.

“Its count of about 110 shell calls per session is its own: round 1 was interrupted and restarted, and the log it read held both attempts, each with 53 to 62 shell calls per candidate.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the training runs that enter round 2, the failure classes are: green_but_graded_fail 30, mangled_nodeids 15, no_edit_seen 4, never_ran_tests 3, base_python_no_pytest 2.

“green_but_graded_fail 30 e.g.['CQCL__guppylang-1259', 'Spenhouet__confluence-↩→ markdown-exporter-92', [...]] mangled_nodeids 15 e.g.['All-Hands-AI__OpenHands-10628', 'davep__braindrop ↩→ -96', [...]] no_edit_seen 4 e.g.['avaiga__taipy-2797', 'bvanelli__actualpy-117', [...]] never_ran_tests 3 e.g.['avaiga__taipy-2797', 'bvanelli__actualpy-117', [...]]”

• Also stated in: § Additional information · plain

  “base_python_no_pytest 2 e.g.['Spenhouet__confluence-markdown-exporter-92', ' ↩→ sissbruecker__linkding-1291', [...]]”

May combine two claims.

4 of 9 observed flips did not reproduce; single-run flip evidence is ~coin flip; conclusions should weight confirmed (twice-run) flips.

“4 of 9 observed flips did not reproduce: single-run flip evidence is ~coin flip; conclusions should weight confirmed (twice-run) flips.”

Classified as probably not a substantial claim. May combine two claims.

The tool classifies all failed training runs in one call and lists tasks that the current harness passes, which the optimizer then uses as regression checks.

“The tool classifies all failed training runs in one call and lists tasks that the current harness passes, which the optimizer then uses as regression checks.”

May combine two claims.

The round-2 edit fixed both earlier regressions and four other validation tasks and broke one, raising validation accuracy from 17 to 22 of 50 tasks.

“The round-2 edit fixed both earlier regressions and four other validation tasks and broke one, raising validation accuracy from 17 to 22 of 50 tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every session still submitted.

“Every session still submitted (good) -keep the anti-deadlock releases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It also added exceptions to the round-1 interpreter rewrite for projects that manage their own environments, a change the notes had listed as a lead.

“It also added exceptions to the round-1 interpreter rewrite for projects that manage their own environments, a change the notes had listed as a lead.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The optimizer wrote corpus triage in its first self-update, after spending 56 to 62 shell calls per candidate on this analysis in round 1.

“The optimizer wrote corpus triage in its first self-update, after spending 56 to 62 shell calls per candidate on this analysis in round 1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two further hooks inject reminders at fixed turns and append rules to the optimizer's system prompt, each citing the round-1 measurement behind it.

“Two further hooks inject reminders at fixed turns and append rules to the optimizer's system prompt, each citing the round-1 measurement behind it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The optimizer also summarized what went wrong in round 1 at the top of its code.

“The optimizer also summarized what went wrong in round 1 at the top of its code.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each cell is one step of a 103-step failed trace from the main corpus of the replay study; the 12 blue steps are the reduced subset whose replay fails exactly the same 14 tests as the full trace.

“Each cell is one step of a 103-step failed trace from the main corpus of the replay study; the 12 blue steps are the reduced subset whose replay fails exactly the same 14 tests as the full trace.”

May combine two claims.

The optimizer could not test its edits before submitting them, so it could not tell which ones were wrong.

“The optimizer could not test its edits before submitting them, so it could not tell which ones were wrong (Appendix J.2 shows the failed edits and why they failed).”

Probably not a substantial claim: may not pass S2 (checkable).

The diagnosis reports make the checks informative.

“The diagnosis reports make the checks informative.”

May combine two claims.

Verification adds execution evidence before submission, and a check whose feedback depends on the cause can lower the overlap and tighten the upper bound.

“Verification adds execution evidence before submission, and a check whose feedback depends on the cause can lower the overlap and tighten the upper bound (Appendix C.1).”

• AssumptionThe authors state this as an assumption: A tool call failure can be caused by different causes (e.g., timeout or incorrect arguments) requiring different repairs.

May combine two claims.

Our analysis identifies conditions under which informative checks can reduce repair-choice error and relates downstream performance to the quality of target selection, repair, and harness selection.

“Our analysis identifies conditions under which informative checks can reduce repair-choice error and relates downstream performance to the quality of target selection, repair, and harness selection.”

May combine two claims.

Its optimizer kept calling the verification tool, but only on tasks that the current harness fails, so none of these calls could show whether an edit breaks a task that already passes.

“Its optimizer kept calling the verification tool, but only on tasks that the current harness fails, so none of these calls could show whether an edit breaks a task that already passes.”

May combine two claims.

The four hosts analyze failed trajectories in different ways, and VERSE improves each of them, so it is not tied to one particular form of failure analysis.

“The four hosts analyze failed trajectories in different ways (Appendix E), and VERSE improves each of them, so it is not tied to one particular form of failure analysis.”

May combine two claims.

This summary shows how well its edits generalize, which training accuracy cannot, because the optimizer has read every training trajectory.

“This summary shows how well its edits generalize, which training accuracy cannot, because the optimizer has read every training trajectory.”

May combine two claims.

Because single runs are this noisy, the tool counts a task as fixed only after two passes.

“Because single runs are this noisy, the tool counts a task as fixed only after two passes.”

Classified as probably not a substantial claim. May combine two claims.

Test accuracy is not available when one chooses a host or a round, so staying above the baseline in every comparison matters in practice.

“Test accuracy is not available when one chooses a host or a round, so staying above the baseline in every comparison matters in practice.”

May combine two claims.

When the recorded evidence is ambiguous, reading trajectories alone leaves a minimum rate of wrong repairs, which executed checks can lower, and wrong repairs limit what self-evolution can gain.

“We prove that when the recorded evidence is ambiguous, reading trajectories alone leaves a minimum rate of wrong repairs, which executed checks can lower, and that wrong repairs limit what self-evolution can gain (Section 3.2).”

May combine two claims.

Self-evolution can act on the remaining terms by choosing more informative checks (lower ρ j ) and using their evidence better (lower ε).

“Self-evolution can act on the remaining terms by choosing more informative checks (lower ρ j ) and using their evidence better (lower ε).”

• AssumptionThe authors state this as an assumption: A tool call failure can be caused by different causes (e.g., timeout or incorrect arguments) requiring different repairs.

Classified as probably not a substantial claim.

Self-written code is loaded only after it passes a safety check and a test run.

“Self-written code is loaded only after it passes a safety check and a test run.”

May combine two claims.

The two factors multiply, so in the bound, better target selection helps little while repairs often fail.

“The two factors multiply, so in the bound, better target selection helps little while repairs often fail (Appendix C.2.2).”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Although the incremental benefit of self-evolution varies across hosts, these findings support a broader principle: harness evolution should improve not only the agent's configuration, but also the procedures that propose, test, and revise it.

“Although the incremental benefit of self-evolution varies across hosts, these findings support a broader principle: harness evolution should improve not only the agent's configuration, but also the procedures that propose, test, and revise it.”

May combine two claims.

Many harness optimizers search and report results on the same benchmark, and such a search resembles test-time scaling, with gains that can come from overfitting to task-specific patterns rather than from a better harness design.

“Many harness optimizers search and report results on the same benchmark (Lee et al., 2026b;Lin et al., 2026;Zhang et al., 2026a;Chen et al., 2026c;Zhang et al., 2026d).Wang et al. (2026b) point out that such a search resembles test-time scaling, and find that its gains can come from overfitting to task-specific patterns rather than from a better harness design.”

• cites Wang et al. (2026b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Everything else is fixed: how harnesses are evaluated and selected, and the turn and execution budgets of each round. Keeping these fixed makes methods comparable under the same protocol.

“Everything else is fixed: how harnesses are evaluated and selected, and the turn and execution budgets of each round.Keeping these fixed makes methods comparable under the same protocol.”

May combine two claims.

The two studies suggest a design: give the optimizer reliable execution checks from the first round, and let it evolve how it uses them.

“Together, the two studies suggest a design: give the optimizer reliable execution checks from the first round, and let it evolve how it uses them.”

May combine two claims.

Our optimizer is a language model and, like other LLM-based optimizers, can also learn from text feedback.

“Our optimizer is a language model and, like other LLM-based optimizers (Yuksekgonul et al., 2025;Agrawal et al., 2026), can also learn from text feedback.”

• cites Yuksekgonul et al., 2025 sourceNot checked — no full text of the source
• cites Agrawal et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

We use three disjoint task sets and measure final performance only on tasks never used during evolution.

“We therefore use three disjoint task sets, as is standard in machine learning (Hastie et al., 2009), and measure final performance only on tasks never used during evolution.”

• cites Hastie et al., 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

A task counts as fixed only if it passes twice, since a single pass can be luck.

“A task counts as fixed only if it passes twice, since a single pass can be luck.”

Classified as probably not a substantial claim. May combine two claims.

We select harnesses by validation accuracy and report the validation-selected harness as our main result, together with the final-round harness.

“We select harnesses by validation accuracy, the percentage of validation tasks a harness resolves.This harness selection mirrors model selection in machine learning, where the checkpoint with the best validation performance is kept rather than the last one (Prechelt, 1998).We report the validationselected harness as our main result, together with the final-round harness.”

• cites Prechelt, 1998Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

After each round, the optimizer is given a short text summary of its validation results, and the optimizer never sees the content of validation tasks or their trajectories.

“After each round, we therefore give it a short text summary of its validation results: its validation accuracy in every round, the identifiers of the validation tasks its latest edit fixed or broke and whether each change held on a rerun, and how often its harness code raised errors.The optimizer never sees the content of validation tasks or their trajectories.”

May combine two claims.

Self-evolution revises g between rounds and can raise r_i and lower e_i in later rounds.

“Self-evolution revises g between rounds and can raise r i and lower e i in later rounds.”

May combine two claims.

Increasing r_i by ∆ raises the lower bound on the success probability of a proposal by only ∆(1-e_i), so better target selection helps little while repairs often fail.

“Increasing r i by ∆ raises the lower bound on the success probability of a proposal by only ∆(1 -e i ), so better target selection helps little while repairs often fail.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Fixed and self-evolving variants with verification share the same probe allowance.

“Fixed and self-evolving variants with verification share the same probe allowance.”

May combine two claims.

These models satisfy the two conditions Wang et al. (2026b) identify for informative harness evolution studies: the executor has headroom, and performance depends strongly on the harness.

“These models satisfy the two conditions Wang et al. (2026b) identify for informative harness evolution studies: the executor has headroom, and performance depends strongly on the harness.”

• cites Wang et al. (2026b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The optimizer never executes validation tasks, but their losses and outcome summaries are returned to it and shape later candidates.

“The optimizer never executes validation tasks, but their losses and outcome summaries are returned to it and shape later candidates.”

May combine two claims.

Feedback can raise r_i and lower e_i in Theorem 2 by showing which edits held up on unseen tasks, and it enlarges a_val through log |F|; restricting its possible values lowers this cost.

“Feedback can raise r i and lower e i in Theorem 2 by showing which edits held up on unseen tasks, and it enlarges a val through log |F|; restricting its possible values lowers this cost.”

May combine two claims.

Three of the 20 Python tasks come from a repository that also appears in the training and validation sets; the in-distribution test set shares no repository with them.

“Three of the 20 Python tasks come from a repository Preprint that also appears in the training and validation sets; the in-distribution test set shares no repository with them.”

Probably not a substantial claim: may not pass S3 (load-bearing).

F_t never contains task statements, tests, or trajectories.

“It never contains task statements, tests, or trajectories.”

May combine two claims.

Theorem 2 (formal): Under equation 9, with ε_sel = E[L(h) - min_{h∈C} L(h)], E L(h) ≤ l0 + (1-l0) ∏_{i=1}^K [1 - r_i(1-e_i)] + ε_sel. If r_i ≥ r and e_i ≤ e for all i, the middle term is at most (1-l0)[1 - r(1-e)]^K.

“Theorem 2 (formal).Under equation 9, with ε sel = E[L( h) -min h∈C L(h)], EL( h) ≤ ℓ 0 + (1 -ℓ 0 ) K i=1 [1 -r i (1 -e i )] + ε sel .(10) If r i ≥ r and e i ≤ e for all i, the middle term is at most (1 -ℓ 0 )[1 -r(1 -e)] K .”

• AssumptionThe authors state this as an assumption: The conditions of Theorem 2 allow proposals to depend on earlier outcomes and on revisions of the optimizer harness; independence between proposals is not required. The repair must lower the population loss, not only fix the training task it was found on.

May combine two claims.

A check can lower the coefficient only if its feedback distinguishes the causes beyond the existing evidence; executing a draft repair provides such feedback.

“A check can lower it only if its feedback distinguishes the causes beyond the existing evidence; executing a draft repair provides such feedback.”

Classified as probably not a substantial claim. May combine two claims.

A validation task whose outcome differs from that of the parent harness is rerun once, and a change that does not reproduce is reverted to the parent's outcome.

“A validation task whose outcome differs from that of the parent harness is rerun once, and a change that does not reproduce is reverted to the parent's outcome.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Self-written code in VERSE is loaded only after a safety check and a test run.

“In our system, the internals of the verification tools are fixed, all optimizer actions are logged and auditable, self-written code is loaded only after a safety check and a test run, the executor acts only inside task containers without network access, and the executor model is frozen, which bounds the scope of self-modification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Grader logs are available on training tasks only.

“Grader logs are available on training tasks only.”

Classified as probably not a substantial claim.

In VERSE, the internals of the verification tools are fixed.

“In our system, the internals of the verification tools are fixed, all optimizer actions are logged and auditable, self-written code is loaded only after a safety check and a test run, the executor acts only inside task containers without network access, and the executor model is frozen, which bounds the scope of self-modification.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All candidates and the optimizer self-update draw checks from one execution budget per round, and every check runs on training tasks.

“All candidates and the optimizer self-update draw checks from one execution budget per round, and every check runs on training tasks.”

May combine two claims.

The comparisons match candidate count, proposal-turn limits, and the evaluation protocol, but not total execution or token cost.

“Thus the comparisons match candidate count, proposal-turn limits, and the evaluation protocol, but not total execution or token cost; Appendix D.3 reports the cost of each run.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Test tasks enter neither loop.

“Test tasks enter neither loop.”

May combine two claims.

The test split was frozen separately, before any method development, and an n-gram overlap scan between the training or validation task texts and the test tasks found no contamination.

“The test split was frozen separately, before any method development, and an n-gram overlap scan between the training or validation task texts and the test tasks found no contamination.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Within each benchmark, all methods use the same K (Section 4.1), so in equation 4 they differ only through r_i, e_i, and ε_sel.

“Within each benchmark, all methods use the same K (Section 4.1), so in equation 4 they differ only through r i , e i , and ε sel .”

May combine two claims.

With no feedback log |F| is 0 and only the choice among T+1 archived harnesses remains; a validation loss per round contributes at most log(m+1); an improved-or-not bit contributes log 2; a list of which tasks changed outcome contributes up to m log 3; and returning validation trajectories would make |F| far larger.

“With no feedback it is 0 and only the choice among T + 1 archived harnesses remains; a validation loss per round contributes at most log(m + 1); an improved-or-not bit contributes log 2; a list of which tasks changed outcome contributes up to m log 3; and returning validation trajectories would make |F| far larger.”

May combine two claims.

The original systems use different executors, edit spaces, optimizer models, acceptance rules, and evaluation protocols.

“The original systems use different executors, edit spaces, optimizer models, acceptance rules, and evaluation protocols.”

May combine two claims.

Lemma C.1.2: β_j ≤ ρ_j β_{j-1}, and β_j = β_{j-1} when κ1 = κ2 at check j.

“Then β j ≤ ρ j β j-1 , and β j = β j-1 when κ 1 = κ 2 at check j.”

Classified as probably not a substantial claim. May combine two claims.

A self-update of the optimizer harness takes effect only if its code passes a safety check and a test run; otherwise the previous version stays in place.

“A self-update of the optimizer harness takes effect only if its code passes a safety check and a test run; otherwise the previous version stays in place.”

Classified as probably not a substantial claim. May combine two claims.

F_t is the validation feedback returned to the optimizer, the same for every method: the validation loss of each round, the identifiers of up to 30 validation tasks whose outcome changed and whether each change held on a rerun, and the rate and two most frequent messages of harness-code errors on validation runs.

“F t is the validation feedback returned to the optimizer, the same for every method: the validation loss of each round, the identifiers of up to 30 validation tasks whose outcome changed and whether each change held on a rerun, and the rate and two most frequent messages of harness-code errors on validation runs.”

Classified as probably not a substantial claim.

Every validation task passed a screening check that its official reference solution passes the grading pipeline.

“Every validation task passed a screening check that its official reference solution passes our grading pipeline.”

Classified as probably not a substantial claim. May combine two claims.

The executor model stays frozen and is evaluated on disjoint task splits.

“The executor model stays frozen and is evaluated on disjoint task splits.”

Classified as probably not a substantial claim.

The executor model in VERSE is frozen.

“In our system, the internals of the verification tools are fixed, all optimizer actions are logged and auditable, self-written code is loaded only after a safety check and a test run, the executor acts only inside task containers without network access, and the executor model is frozen, which bounds the scope of self-modification.”

Classified as probably not a substantial claim. May combine two claims.

Every method runs T = 6 evolution rounds on SWE-rebench and T = 3 on Terminal-Bench, with three candidates generated per round, the same proposal-turn limit, the same sandbox parallelism, and the same harness edit space (executable hooks plus prompts). The only axis that varies between methods is what the optimizer is given: the evidence interface, the verification tools, and the right to edit its

“Every method runs T = 6 evolution rounds on SWE-rebench and T = 3 on Terminal-Bench, with three candidates generated per round, the same proposal-turn limit, the same sandbox parallelism, and the same harness edit space (executable hooks plus prompts).The only axis that varies between methods is what the optimizer is given: the evidence interface, the verification tools, and the right to edit its”

May combine two claims.

VERSE differs from the four baselines in that the optimizer harness is itself optimized, and the execution check is a tool the optimizer calls before submitting an edit, not an outer-loop gate applied afterwards.

“VERSE differs on the two axes this paper studies: the optimizer harness is itself optimized, and the execution check is a tool the optimizer calls before submitting an edit, not an outer-loop gate applied afterwards.”

Classified as probably not a substantial claim. May combine two claims.

Each task is graded by its own test script, and task environments are frozen as image snapshots, so the checker version cannot drift during a run.

“Each task is graded by its own test script, and task environments are frozen as image snapshots, so the checker version cannot drift during a run.”

May combine two claims.

The first term decays geometrically in the number of checks, so a few checks bring the bound close to ε; further gains come from lower ε and higher r_i, which self-evolution can act on.

“The first term decays geometrically in the number of checks, so a few checks bring the bound close to ε; further gains come from lower ε and higher r i , which self-evolution can act on.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All optimizer actions in VERSE are logged and auditable.

“In our system, the internals of the verification tools are fixed, all optimizer actions are logged and auditable, self-written code is loaded only after a safety check and a test run, the executor acts only inside task containers without network access, and the executor model is frozen, which bounds the scope of self-modification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The executor in VERSE acts only inside task containers without network access.

“In our system, the internals of the verification tools are fixed, all optimizer actions are logged and auditable, self-written code is loaded only after a safety check and a test run, the executor acts only inside task containers without network access, and the executor model is frozen, which bounds the scope of self-modification.”

May combine two claims.

When the task domain changes, the executor harness stays specific to its source domain, and what carries over is the optimizer harness learned by self-evolution.

“When the task domain changes, the executor harness stays specific to its source domain, and what carries over is the optimizer harness learned by self-evolution (Section G.2).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Python file is loaded only if it passes the same safety check as the executor's hooks and a test run, so a rejected version leaves the previous one in place.

“the Python file is loaded only if it passes the same safety check as the executor's hooks and a test run, so a rejected version leaves the previous one in place”

Probably not a substantial claim: may not pass S3 (load-bearing).

VERSE seeds only its prompt and candidate guidance with instructions for the verification tools.

“VERSE seeds only its prompt and candidate guidance with instructions for the verification tools,”

May combine two claims.

What crosses domains appears to be the optimizer's methodology, its skills, notes, and tools for diagnosing and verifying failures, while the executor harness stays specific to its source domain.

“What crosses domains appears to be the optimizer's methodology, its skills, notes, and tools for diagnosing and verifying failures, while the executor harness stays specific to its source domain.”

Classified as probably not a substantial claim. May combine two claims.

A closed-weight LLM supervisor, a different model from the executor, produces every diagnosis, and every claim is checked by re-executing the trace in the task's Docker environment against the task's own tests.

“A closed-weight LLM supervisor, a different model from the executor, produces every diagnosis, and every claim is checked by re-executing the trace in the task's Docker environment against the task's own tests.”

May combine two claims.

Harness components that fix tool use errors common across models, such as shell quoting, transfer; components tuned to one model's behavioral quirks do not.

“The contrast suggests a mechanism: harness components that fix tool use errors common across models, such as shell quoting, transfer; components tuned to one model's behavioral quirks do not.”

Classified as probably not a substantial claim. May combine two claims.

Before each tool call, one hook caps the verification calls of each candidate session so that parallel sessions share the pool, caps exploratory shell calls, and blocks a submission once if no draft has been verified yet.

“Before each tool call, one hook caps the verification calls of each candidate session so that parallel sessions share the pool, caps exploratory shell calls, and blocks a submission once if no draft has been verified yet.”