VeRO: A Harness for Agents to Optimize Agents
Open accessAbstract
An important emerging application of coding agents is agent harness optimization: the iterative improvement of a target agent by editing and evaluating its code. Despite its relevance, the community lacks a systematic understanding of coding agent performance on this task. Harness optimization differs from conventional software engineering: agent harnesses interleave deterministic code with stochastic LLM completions, requiring structured capture of both intermediate execution traces and downstream outcomes. To address these challenges, we introduce (1) VeRO (Versioning, Rewards, and Observations), an outer harness that provides versioned snapshots, budget-controlled evaluation, and structured execution traces of target harnesses, and (2) VeRO-Bench, a benchmark suite of target agents and tasks with reference evaluation procedures. Using VeRO, we conduct an empirical study comparing optimizers across tasks and analyzing which modifications reliably improve target agent harnesses. We release VeRO to support research on agent optimization as a core capability for coding agents. Code is available at https://github.com/scaleapi/vero.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit.
“A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit.”
From HarnessOpt-Bench: Evaluating LLMs at Harness Optimization (2026) · cites this paper as [23]
The Permission control and Experiment Database sections describe enforced evaluation restrictions and a gated runner that counts evaluation calls. The Git Worktree section describes an immutable commit history that makes changes auditable. The paper does not attribute these functions to a trusted execution environment, and its A.4 Limitations section says the budget does not measure broader resource use such as tokens or API costs. (LLM, conf=0.88)
Experimental — accuracy not yet measuredMethods are today evaluated with their own target agents, seeds, budgets, disclosure policies, and scoring protocols, so their results conflate the optimizer model, the coding harness it acts through, the target agent, and the protocol.
“Methods are today evaluated with their own target agents, seeds, budgets, disclosure policies, and scoring protocols, so their results conflate the optimizer model, the coding harness it acts through, the target agent, and the protocol.”
From HarnessOpt-Bench: Evaluating LLMs at Harness Optimization (2026) · cites this paper as [23]
Section 3.2, “A Protocol for Benchmarking Coding Agents as Agent Optimizers” (page unavailable), says meaningful comparisons require identical resource and environment conditions. The Protocol section (page unavailable) also distinguishes the coding-agent scaffold from the optimizer model. But the paper does not establish the broader claim that methods today each use different target agents, seeds, disclosure policies, and scoring protocols, or that their reported results consequently conflate all four named factors. (LLM, conf=0.82)
Experimental — accuracy not yet measuredUrsekar et al. and Lee et al. let a coding agent read the target source, inspect scores and execution traces from prior candidates, and choose what evidence to gather, rather than orchestrating mutations in a fixed search algorithm.
“Closest to the regime HARNESSOPT-BENCH measures are optimizers that act end-to-end: Ursekar et al. [23] and Lee et al. [12] let a coding agent read the target source, inspect scores and execution traces from prior candidates, and choose what evidence to gather, rather than orchestrating mutations in a fixed search algorithm.”
From HarnessOpt-Bench: Evaluating LLMs at Harness Optimization (2026) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 26
- J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, and et al. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. · arXiv
- J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, A. Madry, and L. Weng. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=6s5uXNWGIh.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, and et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
- A. Cheng, A. Jacovi, A. Globerson, B. Golan, C. Kwong, and et al. The facts leaderboard: A comprehensive benchmark for large language model factuality, 2025. URL https://arxiv.org/abs/2512.10791. · arXiv
- C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the new autodiff -unlocking efficient optimization of computational workflows. In Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs, 2024. URL https://openreview.net/forum?id=Lg4AnAFsMd.
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, and et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, and et al. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- S. Hu, C. Lu, and J. Clune. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=t9U3LW7JVX.
- N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, and et al. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL.
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, and et al. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.
- O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, and et al. DSPy: Compiling declara- tive language model calls into self-improving pipelines. In The Twelfth International Conference on Learning Representations, 2024.
- G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. Gaia: a benchmark for general ai assistants, 2023. URL https://arxiv.org/abs/2311.12983. · arXiv
- A. Novikov, N. V ũ, M. Eisenberger, E. Dupont, P.-S. Huang, and et al. Alphaevolve: A coding agent for scientific and algorithmic discovery, 2025. URL https://arxiv.org/abs/2506.13131. · arXiv
- D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, and et al. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= Ti67584b98.
- M. Robeyns, M. Szummer, and L. Aitchison. A self-improving coding agent, 2025. URL https://arxiv.org/ abs/2504.15228. · arXiv
- L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
- X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, and et al. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. URL https://arxiv.org/abs/2407.16741. · arXiv
- J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, and et al. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368. · arXiv
- C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= Bb4VGOWELI.
- J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, and et al. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2405.15793. · arXiv
- S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, and et al. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
- S. Yao, N. Shinn, P. Razavi, and K. R. Narasimhan. τ-bench: A benchmark for Tool-Agent-User interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=roNSXZpUDN.
- L. Yin and Z. Wang. Llm-autodiff: Auto-differentiate any llm workflow, 2025. URL https://arxiv.org/abs/ 2501.16673.
- E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. In Conference on Language Modeling (CoLM), 2024. URL https://arxiv.org/abs/2310. 02304.
- J. Zhang, J. Xiang, Z. Yu, F. Teng, X.-H. Chen, and et al. AFlow: Automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=z5uVAKwmjf.
- J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune. Darwin gödel machine: Open-ended evolution of self- improving agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=pUpzQZTvGY.
Cited by
Sorted by citations
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Ursekar, V., Shanker, A., Maurya, Y., Yasser, S., Kalmath, V. S., Chatrath, V., & Xue, Y. (2026) · arXiv · doi:10.48550/arxiv.2608.06301
“A trusted execution environment, building on V E RO [23], enforces access and target-evaluation budgets, isolates held-out state, meters resource use, and versions every candidate for audit (Figure 1).”
Self-citationAutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Park, S., Kim, W., Tan, R., Zhang, J., Han, W.-S., Gao, P., Park, C., Yao, Y., Fu, R., Nallipogu, E., et al. (2026) · arXiv · doi:10.48550/arxiv.2608.23041
Evo-Bench: Can Language Models Improve Agent Harness?
Huang, L., Yang, C., Zhou, H., Song, H., Chen, Z., Le, R., Song, Y., Zhao, W. X., & Zhang, T. (2026) · arXiv · doi:10.48550/arxiv.2608.09096
VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
Wang, Z., Ge, Y., Wang, Z., Wang, H., Xu, Y., Frandsen, J., Fu, S., Wilson, A. C., & Reddy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.02616
- Authors
- Ursekar, V., Shanker, A., Chatrath, V., Xue, Y., & Denton, S. M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2602.22480
- OpenAlex
- W7131850018
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- coding agents, large language models, execution traces, version control, benchmark suite
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
