CoChat
Sign Up
PreprintCited by 1AI-based Problem Solving and PlanningEvolutionary Algorithms and ApplicationsArtificial Intelligence Applications

Harness^2: Recursive Agent Harnessing for an Open World

Open access
Xu, Ruiyao, Chen, Yanfei, CuiZhu, Zhongying, Mishra, Bhavana Dalvi, Ming, Yifei, Yu, Han, Han, Rujun, Lee, Chen-Yu +1 more·2026·arXiv
DOI 10.48550/arxiv.2610.14224arXiv 2610.14224
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM agents' capabilities depend not only on their underlying models but also on their execution harnesses. Recursive self-improvement (RSI) offers a promising approach to automatically evolve these harnesses. However, deploying RSI in open-world settings still faces two critical challenges. First, open-world tasks are typically unseen during the harness design phase and novel at test time, leaving no ground-truth verifiers to reliably guide online harness edits. Second, the RSI process remains highly task-specific and fails to generalize to novel scenarios. We introduce Harness^2, a plug-and-play, recursive agent harnessing framework that addresses these challenges through two connected levels of recursion. Task-level recursion refines the current task's harness using a contrastive Propose-Probe-Compose step that reliably guides edits without ground-truth verifiers, and can be scaled further via parallel or sequential recursion. Harnessing-level recursion then extracts experience from these edits to update a continual improvement harness, enabling the agent to better adapt to future tasks. Across three professional benchmarks and four model-harness configurations, a single step consistently improves GLM-5 and Gemini 3.5 Flash over their base harnesses, yielding gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB, with further gains as the task-level recursion budget increases.

AI snapshot

Written from the paper’s full text.

TL;DR

Harness^2 is a plug-and-play recursive agent harnessing framework that refines a task’s execution harness through a contrastive Propose-Probe-Compose step and then updates an improvement harness from that refinement experience. Across three professional benchmarks and four model-harness configurations, one improvement step and parallel recursion improve GLM-5 and Gemini 3.5 Flash over their base harnesses.

Key findings

  • A single contrastive improvement step improves all three benchmarks in every configuration, with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB Partial pass rate.
  • Parallel task-level recursion at \(k=5\) adds further gains over a single step: up to 3.9 points on WorkBuddy-Bench, 4.2 points on JobBench, and 6.7 percentage points on LAB Partial.
  • Ablations show all three components contribute: removing the library lowers WorkBuddy-Bench and JobBench by 3.1 and 1.8 points; removing scoping costs 2.9 and 3.7; removing composition costs 1.6 and 1.9.
  • With the same model as executor and editor, all five tested models improve on every benchmark; Opus 5 gains 3.3, 5.5, and 3.0 points on WorkBuddy-Bench, JobBench, and LAB, while GLM-5 gains 13.7 points on LAB. Tool-error rates fall on all three benchmarks, most on JobBench (14.4% → 9.1%).

Methods

  • Design: Harness^2 couples task-level recursion, which refines the current execution harness via Propose-Probe-Compose, parallel recursion, or sequential recursion, with harnessing-level recursion, which updates a continual improvement harness and its library of adopted harnesses plus EXPERIENCE.md entries. The base model is frozen; the harness has eight editable components: system prompt, guardrails, memory, skills, tool guidance, scripts, subagents, and a middleware plugin.
  • Data: WorkBuddy-Bench (Code 80, Web 70, Office 50 tasks; test halves 40/35/25), JobBench (65 tasks, frozen 31/34 split, 34 test tasks), and LAB (1,251 legal tasks; 100 held-out test tasks, 24 dev tasks). Executors are Gemini 3.5 Flash and GLM-5 under OpenCode and Codex; the editor and selector are Claude Opus 4.8. Baselines include Base harness, ReasoningBank, Meta-Harness, and Harness Scaling.
  • Analysis: Main tables report means and standard deviations over three runs with shuffled task order; additional analyses cover scaling budget \(k\), compute-performance trade-off, ablations, library transfer, same-model executor/editor, tool-use reliability, and case studies.

Limitations

  • Paper states: recursive refinement requires additional executions and editor sessions, so increasing the recursion budget adds inference cost; parallel sessions can reduce latency when resources permit but do not eliminate the extra computation, and experiments characterize a bounded range of budgets rather than a universally optimal allocation.
  • Paper states: without ground-truth supervision, refinement relies on behavioral contrasts and model judgment, which remain sensitive to execution variability and shared model biases and do not guarantee correctness or transfer to a different task.
  • Text makes evident: failure analysis reports selection can prefer a weaker candidate (e.g., on WB-code an output scoring 55 chosen over one scoring 95; on LAB a narrower-coverage candidate reduces the score from 54 to 26), incorrect premises can propagate across candidates, and missing judge verdicts or outcome variability complicate attribution; the leave-one-out library reuse test is an offline reuse analysis, and repeated execution may contribute to differences.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2610.14224

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 18 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Open-world tasks typically lack ground-truth labels or reliable verifiers.

    “tasks typically lack ground-truth labels or reliable verifiers.”
    • Also stated in: § 2.1. Recursive Agent Harnessing · plain
      “without ground-truth answers or labeled validation feedback.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Lee et al., 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu et al., 2026 source

      The Benchmark and Evaluation Details section says the evolver receives outcome labels only after tasks resolve, supporting a narrower point: labels can be unavailable when a task arrives. It also describes official outcomes and verification for the studied benchmarks. The paper does not establish that open-world tasks typically lack ground-truth labels or reliable verifiers. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
    • cites Nie et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Open-world tasks are typically unseen during the harness design phase and novel at test time.
  • Findingown data

    Only 2-10% of modified files retain at least half their lines from a library entry.

    “Only 2-10% of modified files retain at least half their lines from an entry”
    Quote in the paperNot a substantial claim
  • Findingown data

    Every ablated harness still exceeds the base.

    “Every ablated harness still exceeds the base.”
    Quote in the paperNot a substantial claim
  • Findingown data

    A single contrastive improvement step improves GLM-5 and Gemini 3.5 Flash over their base harnesses across three professional benchmarks and four model-harness configurations.

    “a single step consistently improves GLM-5 and Gemini 3.5 Flash over their base harnesses”
    • Also stated in: § 1. Introduction · plain
      “One contrastive step improves performance on all three benchmarks in every configuration”
    • Also stated in: § 3.1. Experiment Setup · plain
      “A single improvement step improves both GLM-5 and Gemini 3.5 Flash under OpenCode and Codex.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.
  • Findingown data

    Harness 2 delivers consistent gains across models and benchmarks.

    “Harness 2 delivers consistent gains across models and benchmarks.”
    • Also stated in: § 1. Introduction · boosted
      “We demonstrate consistent gains across three benchmarks and four executor-harness configurations”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 61
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    WorkBuddy-Bench covers office-style deliverables in three domains (Code, Web, Office).

    “WorkBuddy-Bench (Team et al., 2026) covers office-style deliverables in three domains (Code, Web, Office).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Team et al., 2026 source

      Our copy does not mention WorkBuddy-Bench or describe its domains. The benchmark discussion in §6.2 concerns CC-Bench-V2 and frontend, backend, and long-horizon engineering tasks; searches for “WorkBuddy-Bench” and “office-style deliverables Code Web Office” found no passage addressing the claimed three-domain coverage. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Offline methods search over candidate harnesses using reusable training tasks and evaluation feedback.

    “Offline methods search over candidate harnesses using reusable training tasks and evaluation feedback (Chen et al., 2026;Zhang et al., 2026d,e).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Chen et al., 2026
    • cites Zhang et al., 2026d,e

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    On WorkBuddy-Bench, a single step lifts GLM-5 to 75.7 with OpenCode and 75.3 with Codex, surpassing the corresponding base Gemini scores of 72.4 and 74.5.

    “On WorkBuddy-Bench, a single step lifts GLM-5 to 75.7 with OpenCode and 75.3 with Codex, surpassing the corresponding base Gemini scores of 72.4 and 74.5.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Both parallel and sequential recursion improve monotonically as k increases from 1 to 5 on WB-code and WB-office.

    “Both improve monotonically as k increases from 1 to 5.”
    • Also stated in: § 3.1. Experiment Setup · plain
      “Parallel recursion rises from 76.5 to 80.8 on Code and from 79.5 to 83.4 on Office.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Library provenance is present in 22% of modified files on LAB and WB-office and 47% on JobBench.

    “Library provenance is present in 22% of modified files on LAB and WB-office and 47% on JobBench.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Parallel recursion finishes above sequential recursion at k=5, although sequential recursion is higher on Code at k=2.

    “Parallel recursion finishes above sequential recursion at k=5, although sequential recursion is higher on Code at k=2.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    The single-step gains reach up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB Partial pass rate.

    “yielding gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB”
    • Also stated in: § 1. Introduction · plain
      “with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points in LAB Partial pass rate.”
    • Also stated in: § 3.1. Experiment Setup · plain
      “On WorkBuddy-Bench, Harness 2 raises GLM-5's average score by 11.5 points under OpenCode, the largest one-step improvement over the base harness on this benchmark.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Harness 2 exceeds the compared baselines on WorkBuddy-Bench Avg. and JobBench Score in all four configurations; on LAB Partial, Meta-Harness leads for Gemini 3.5 Flash under Codex.

    “Harness 2 exceeds the compared baselines on WorkBuddy-Bench Avg. and JobBench Score in all four configurations; on LAB Partial, Meta-Harness leads for Gemini 3.5 Flash under Codex.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Harness Scaling regresses from k=2 to 3 on both domains, while ReasoningBank fluctuates.

    “Harness Scaling regresses from k=2 to 3 on both domains, while ReasoningBank fluctuates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ReasoningBank and Harness Scaling yield smaller gains, while Meta-Harness uses more compute than Harness 2 at k=5 for lower performance.

    “ReasoningBank and Harness Scaling yield smaller gains, while Meta-Harness uses more compute than Harness 2 at k=5 for lower performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On WorkBuddy-Bench, Gemini's gap falls from 2.1 to 0.4, whereas GLM-5's increases from 0.2 to 1.3.

    “On WorkBuddy-Bench, Gemini's gap falls from 2.1 to 0.4, whereas GLM-5's increases from 0.2 to 1.3.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    The same pattern extends to JobBench and LAB, where gains reach 4.5 points and 11.8 percentage points in Partial pass rate, respectively.

    “The same pattern extends to JobBench and LAB, where gains reach 4.5 points and 11.8 percentage points in Partial pass rate, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Removing the library lowers WorkBuddy-Bench and JobBench scores by 3.1 and 1.8 points, respectively; removing scoping costs 2.9 and 3.7, and removing composition costs 1.6 and 1.9.

    “Table 3 shows that removing the library lowers WorkBuddy-Bench and JobBench scores by 3.1 and 1.8 points, respectively; removing scoping costs 2.9 and 3.7, and removing composition costs 1.6 and 1.9.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Removing skills causes the largest loss on Code and Web (3.6 and 6.9 points), while removing scripts has the largest effect on Office (7.4 points).

    “Removing skills causes the largest loss on Code and Web (3.6 and 6.9 points), while removing scripts has the largest effect on Office (7.4 points).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On JobBench, Gemini gains 4.5 and 1.4 points under OpenCode and Codex, compared with GLM-5's 3.6 and 1.2.

    “On JobBench, Gemini instead gains 4.5 and 1.4 points under OpenCode and Codex, compared with GLM-5's 3.6 and 1.2.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Both recursive forms of Harness 2 at k=2 outperform every plotted baseline at k=5 on both domains.

    “Notably, both recursive forms of Harness 2 at k=2 outperform every plotted baseline at k=5 on both domains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Parallel recursion rises from 76.5 to 80.8 on Code and from 79.5 to 83.4 on Office.

    “Parallel recursion rises from 76.5 to 80.8 on Code and from 79.5 to 83.4 on Office.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Opus 5 gains 3.3, 5.5, and 3.0 points on WorkBuddy-Bench, JobBench, and LAB; GLM-5 gains 13.7 points on LAB.

    “Opus 5 gains 3.3, 5.5, and 3.0 points on WorkBuddy-Bench, JobBench, and LAB; GLM-5 gains 13.7 points on LAB.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Code scores drop from 73.4 to 63.2 and Web scores fall from 69.9 to 65.0 when directly refining without access.

    “Specifically, Code scores drop from 73.4 to 63.2 and Web scores fall from 69.9 to 65.0.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    ReasoningBank and Harness Scaling show smaller or inconsistent gains as k increases.

    “In contrast, ReasoningBank and Harness Scaling show smaller or inconsistent gains as k increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On WorkBuddy-Bench, Harness 2 raises GLM-5's average score by 11.5 points under OpenCode, the largest one-step improvement over the base harness on this benchmark.

    “On WorkBuddy-Bench, Harness 2 raises GLM-5's average score by 11.5 points under OpenCode, the largest one-step improvement over the base harness on this benchmark.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    On JobBench, the initial OpenCode-Codex gaps of 2.4 points for Gemini and 3.0 for GLM-5 both shrink to 0.5.

    “On JobBench, the initial OpenCode-Codex gaps of 2.4 points for Gemini and 3.0 for GLM-5 both shrink to 0.5.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • May combine two claims.

    Findingown data

    Harness2's gains over its own starting harness are 7.4, 8.4, and 9.4 points on Code, Web, and Office.

    “The corresponding gains over its own starting harness are 7.4, 8.4, and 9.4 points.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A harness refined for Gemini 3.5 Flash improves all three other executors on both Code and Office.

    “All three improve on both Code and Office.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Gemini 3.7 Flash gains 0.3 on Code and 6.4 on Office.

    “Gemini 3.7 Flash gains 0.3 on Code and 6.4 on Office.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    In the WorkBuddy-Bench Office case, a guardrail treats supplied inputs as read-only and applies the stated cleaning rules before aggregating.

    “a guardrail treats inputs as read-only and applies the stated cleaning rules before aggregating”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    In the improved harness for WorkBuddy-Bench Web, preview, errors and export are re-derived together.

    “✓ Preview, errors and export re-derived together”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    No fixed harness leads on every domain: Claude Code performs best on Code and Office, while Codex leads on Web.

    “No fixed harness leads on every domain: Claude Code performs best on Code and Office, while Codex leads on Web”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On WorkBuddy-Bench, removing the library lowers the score from 77.5 to 74.4, removing scoping to 74.6, and removing composition to 75.9.

    “| w/o library | 74.4 | 34.2 |”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “| w/o scoping | 74.6 | 32.3 |”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “| w/o composition | 75.9 | 34.1 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    In the WorkBuddy-Bench Web case, a guardrail forbids network dependencies.

    “a guardrail forbids network dependencies”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The optimized re-run successfully cross-referenced the figures and restored the memo's missing sections.

    “The optimized re-run successfully cross-referenced the figures and restored the memo's missing sections”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the WorkBuddy-Bench Web case, a script checks that the artifact opens offline.

    “a script checks the artifact opens offline”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the JobBench Social Science RAS task3 case, the composed harness adds, among other edits, a plugin that appends a warning to tool results after a fixed number of calls without an artifact, a subagent that returns citable tables in its own context, and durable domain memory, raising the score from 72.0 to 100.0.

    “the composed harness adds, among other edits, a plugin that appends a warning to tool results after a fixed number of calls without an artifact, a subagent that returns citable tables in its own context, and durable domain memory (72.0 → 100.0).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After one improvement step, error rates fall on all three benchmarks, most on JobBench (14.4% → 9.1%).

    “Error rates fall on all three benchmarks, most on JobBench (14.4% → 9.1%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the LAB arbitration case, the base harness leaves markup in Word's track-changes layer, and when converted to text, the markup vanishes.

    “1 Markup left in Word's track-changes layer 2 Converted to text, the markup vanishes”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On WorkBuddy-Bench (Gemini 3.5 Flash, OpenCode), Harness 2 (full) achieves an average of 80.8 versus 72.4 for the base harness.

    “| Base harness | 73.4 | 69.9 | 74.0 | 72.4 |”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “| Harness 2 (full) | 80.8 | 78.3 | 83.4 | 80.8 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The current method specifies the recursion form and budget in advance.

    “We currently specify the recursion form and budget in advance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the improved harness for WorkBuddy-Bench Office, every channel is found and all checks pass.

    “✓ Every channel found; all checks pass”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Opus 4.8 gains 2.6 and 1.1 points.

    “Opus 4.8 gains 2.6 and 1.1 points.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The number of tool calls decreases on WorkBuddy-Bench but increases on JobBench and LAB after one improvement step.

    “The number of calls decreases on WorkBuddy-Bench but increases on JobBench and LAB.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    GLM-5 gains 17.3 points on Code (58.5 → 75.8) when deployed with a harness refined for Gemini 3.5 Flash.

    “GLM-5 gains 17.3 points on Code (58.5 → 75.8)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The current improvement harness retains adopted harnesses and experience in a simple indexed library.

    “The current improvement harness retains adopted harnesses and experience in a simple indexed library.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    In the JobBench Social Science RAS task3 case, with Claude Opus 5 as executor and editor, research expands to fill the budget and files stay unwritten near the deadline.

    “With Claude Opus 5 as executor and editor, research expands to fill the budget and files stay unwritten near the deadline”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Harness 2 makes fewer tool calls on WorkBuddy-Bench and more on JobBench and LAB.

    “it makes fewer calls on WorkBuddy-Bench and more on JobBench and LAB.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On JobBench, removing the library lowers the score from 36.0 to 34.2, removing scoping to 32.3, and removing composition to 34.1.

    “| w/o library | 74.4 | 34.2 |”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “| w/o scoping | 74.6 | 32.3 |”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “| w/o composition | 75.9 | 34.1 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In the WorkBuddy-Bench Web case, page state is frozen at first render, so remapping and export go stale.

    “Page state is frozen at first render, so remapping and export go stale”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the WorkBuddy-Bench Office case, a skill pins the grain and the derived fields.

    “a skill pins the grain and the derived fields”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For GLM-5 under OpenCode, Harness 2 reports higher JobBench and LAB scores than the base harness.

    “Base harness | 20.6 (2.0) 20.0 (0.8) 59.6 (2.7) 13.6 (0.3) 23.6 (2.3) 22.3 (2.1) 58.1 (2.3) 16.5 (0.7)”
    • Also stated in: § # WHAT YOU DO NOT HAVE · plain
      “Harness 2 | 24.2 (0.9) 23.6 (1.4) 71.4 (2.9) 15.8 (2.1) 24.8 (0.2) 25.3 (1.4) 65.9 (2.7) 22.3 (2.5)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After the stream completes, re-running each task with its own entry removed from the library raises scores from 77.5 to 79.4 on WorkBuddy-Bench and from 36.0 to 37.7 on JobBench.

    “Scores rise from 77.5 to 79.4 on WorkBuddy-Bench and from 36.0 to 37.7 on JobBench.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    In the WorkBuddy-Bench Office case, the reference harness aggregates at the wrong grain and no channel row matches the key.

    “The reference aggregates at the wrong grain and no channel row matches the key”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Lessons are retained only when supported by execution contrasts; selecting an output alone does not establish transferability.

    “Lessons are retained only when supported by execution contrasts; selecting an output alone does not establish transferability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Increasing test-time exploration alone does not ensure better performance; recursive harness refinement produces more consistent gains in these settings.

    “Increasing test-time exploration alone therefore does not ensure better performance; recursive harness refinement produces more consistent gains in these settings.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

  • Classified as probably not a substantial claim.

    Premisecitation

    Sequential recursion applies iterative self-refinement to the execution harness.

    “Sequential recursion applies iterative self-refinement (Madaan et al., 2023) to the execution harness.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Madaan et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Reasoning-Bank keeps the harness fixed and retrieves distilled strategies from past trajectories back into context.

    “Reasoning-Bank (Ouyang et al., 2026b) keeps the harness fixed and retrieves distilled strategies from past trajectories back into context.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ouyang et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    These contrasts inform subsequent composition rather than verify task correctness, as execution stochasticity and shared model biases limit the reliability of the evidence.

    “These contrasts inform subsequent composition rather than verify task correctness, as execution stochasticity and shared model biases limit the reliability of the evidence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Main tables report means and standard deviations over three runs, each with a shuffled task order.

    “Main tables report means and standard deviations over three runs, each with a shuffled task order”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LAB is a legal-work benchmark; following its protocol we report Partial, the per-criterion pass rate.

    “LAB (Harvey AI, 2026) is a legal-work benchmark; following its protocol we report Partial, the per-criterion pass rate.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Harvey AI, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Meta-Harness searches harness code offline against a training split and deploys the best one.

    “Meta-Harness (Lee et al., 2026b) searches harness code offline against a training split and deploys the best one.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lee et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The improvement harness remains fixed during task-level recursion and is updated afterward through its harness improvement library.

    “The improvement harness remains fixed during task-level recursion and is updated afterward through its harness improvement library.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Harness Scaling revises the harness for each evaluation instance by spending additional inference budget on revision without carrying state across instances.

    “Harness Scaling (Wang et al., 2026a) revises the harness for each evaluation instance by spending additional inference budget on revision without carrying state across instances.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al., 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationnone given

    Harness2 combines contrastive task-level refinement with updates to the improvement harness in a one-pass deployment stream, without ground-truth supervision or model updates.

    “Harness 2 combines contrastive task-level refinement with updates to the improvement harness in a one-pass deployment stream, without ground-truth supervision or model updates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The ablation results support contributions from multiple component groups, but do not establish additive effects.

    “The results support contributions from multiple component groups, but do not establish additive effects.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Model weights, stage instructions, and the decision rule remain fixed, while the library evolves.

    “Model weights, stage instructions, and the decision rule remain fixed, while the library evolves.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    JobBench targets end-to-end professional job tasks with long executions.

    “JobBench (Li et al., 2026) targets end-to-end professional job tasks with long executions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The reference execution reveals how the harness behaves, not whether its deliverable is correct.

    “The reference reveals how the harness behaves, not whether its deliverable is correct.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisereasoning

    Read as text, a finished .docx shows only the accepted text because native tracked changes collapse and comments are dropped, so a redline built only from them reads as a clean rewrite.

    “Read as text, the finished .docxshows only the accepted text: native tracked changes collapse and comments are dropped, so a redline built only from them reads as a clean rewrite.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These signals help assess the effects of a proposed change, but remain sensitive to execution variability and shared model biases.

    “These signals help assess the effects of a proposed change, but remain sensitive to execution variability and shared model biases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The leave-one-out re-run can access entries from both earlier and later tasks, so it is an offline reuse analysis rather than an estimate of online gains.

    “The re-run can access entries from both earlier and later tasks, so this is an offline reuse analysis rather than an estimate of online gains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The rubric is never shown to the executor, the editor, or the selector.

    “The rubric is never shown to the executor, the editor, or the selector.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A guard refuses to hand over any working directory that contains a grader verdict, rubric, or score.

    “A guard refuses to hand over any working directory that contains a grader verdict, rubric, or score.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Every LAB number in the paper is on the 100 test tasks, which each method processes as one stream; the 24 dev tasks are used only by Meta-Harness as its search set.

    “Every LAB number in this paper is on the 100 test tasks, which each method processes as one stream; the 24 dev tasks are used only by Meta-Harness as its search set.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Refinement can improve performance without shortening execution, distinguishing more reliable tool use from fewer calls.

    “These results distinguish more reliable tool use from fewer calls; refinement can improve performance without shortening execution.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The comparison with production harnesses shows the benefit of refining an existing harness, while noting that refinement uses additional test-time compute.

    “This comparison shows the benefit of refining an existing harness, while noting that refinement uses additional test-time compute.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The verifier never exposes its expected values to any model in the pipeline.

    “The verifier never exposes its expected values to any model in our pipeline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Meta-Harness and MemoHarness retain search or diagnostic experience for later edits.

    “Meta-Harness and MemoHarness retain search or diagnostic experience for later edits (Huang et al., 2026b;Lee et al., 2026b)”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Huang et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lee et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The scores used in the qualitative case studies are retrospective evaluations and are not available to the pipeline.

    “The scores are retrospective evaluations and are not available to the pipeline.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Ablation and transfer studies demonstrate the value of scoped proposals, composition, and accumulated harnessing experience.

    “Ablation and transfer studies further demonstrate the value of scoped proposals, composition, and accumulated harnessing experience.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each reported score is the mean over three independent streams.

    “Each reported score is the mean over three independent streams”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Test-time methods refine a task-specific harness through execution feedback or carry an evolving harness across unlabeled batches.

    “Test-time methods refine a task-specific harness through execution feedback or carry an evolving harness across unlabeled batches (Lee et al., 2026a;Nie et al., 2026b;Wang et al., 2026a).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Lee et al., 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Nie et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All baselines use the same frozen executor and initial harness, process the same test stream, and are evaluated by the same judge.

    “All baselines use the same frozen executor and initial harness, process the same test stream, and are evaluated by the same judge.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    These signals do not guarantee correctness or establish that an adopted change will transfer to a different task.

    “They do not guarantee correctness or establish that an adopted change will transfer to a different task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Parallel sessions can reduce latency when resources permit, but do not eliminate the additional computation required by Harness 2.

    “Parallel sessions can reduce latency when resources permit, but do not eliminate this additional computation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Adopted harnesses can transfer across models, although the benefit varies substantially with the executor and domain.

    “The gains show that adopted harnesses can transfer across models, although the benefit varies substantially with the executor and domain.”
    Quote in the paperSubstantial claim
83 of 83

  • The paper relies on this without support.

    The three benchmarks (WorkBuddy-Bench, JobBench, LAB) adequately represent open-world professional tasks.

    “We evaluate on three benchmarks from distinct professional domains.”
    “Across three professional benchmarks and four model-harness configurations”

  • The paper relies on this without support.

    Open-world tasks are typically unseen during the harness design phase and novel at test time.

    “open-world tasks are typically unseen during the harness design phase and novel at test time”

  • The paper relies on this without support.

    The three professional benchmarks and their test streams are representative of open-world deployment tasks.

    “Together, these findings support recursive harnessing as a way to turn task-specific refinement into a reusable foundation for online test-time adaptation across open-world tasks.”

  • The paper relies on this without support.

    The LLM judges used for grading produce valid task scores.

    “Deliverables are judged against a weighted rubric by Claude Opus 4.8; the task score is the total weighted score over the maximum score, which is the benchmark's own metric.”

  • The paper relies on this without support.

    The reference solutions against which runs are evaluated are correct.

    “When evaluated against the reference”
5 of 5

Cites

Flagged first, then by citations

5 of the 48 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. V. A, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=sY5N0zY5Od.

Cited by

Sorted by citations

Authors
Xu, R., Chen, Y., CuiZhu, Z., Mishra, B. D., Ming, Y., Yu, H., Han, R., Lee, C.-Y., & Pfister, T.
Affiliations
  • Northwestern University (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2610.14224
arXiv
2610.14224
OpenAlex
W7221780331
Semantic Scholar
6fac819b95eb96e9bf74c44e8fe102e48b084d73
Cited by
1
References
48
Citations by year
2026: 1
Type
posted content
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
recursive self-improvement, LLM agents, agent harnesses, continuous improvement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review