Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

In the paper

“As shown in Table 1, HERMES improves the resolved rate for 19 of the 20 backbones, ties on the remaining one (Claude Opus 5), and outperforms both baselines for the two backbones evaluated against two harnesses.”

In the paper

“With GPT5.6 Sol, the composite score rises from 6.5% to 31.0% under medium effort ( +24 . 5 ) and from 19.0% to 36.5% under high effort ( +17 . 5 )”

In the paper

“Claude Opus 5 still gains 8.0 and 7.0 points”

In the paper

“with Claude Sonnet 5, HERMES improves resolution by 6.2 points while reducing average latency from 962.37 s to 824.63 s”

In the paper

“HERMES reaches 97.0% with GPT-5.6 Sol against 96.2% under mini-SWE-agent.”

In the paper

“Luna gain 5.04 and 5.68 points over their respective baseline harnesses”

In the paper

“Most configurations consume additional tokens and cost, but Claude Sonnet 5 is a notable exception: HERMES improves its resolution while reducing both token consumption and inference cost at either effort level.”

In the paper

“The exception is Claude Opus 5, whose baseline configurations are already strong and gain 1.5 and 1.2 points.”

In the paper

“Recovering those 2.1 points costs a further $1.27k, split between upgrading diagnosis ( +0 . 9 points for $0.38k) and upgrading the Dev-Primitives ( +1 . 2 points for $0.89k).”

In the paper

“on TerminalBench, using GPT-5.5 only for diagnosis improves resolution from 27.6% to 46.1% ( +18 . 5 ) against 36.7% ( +9 . 1 ) for activation alone”

In the paper

“As shown in Table 3, HERMES consistently improves resolution under matched model-effort configurations, by 18.8 and 18.2 points for GPT-5.6 Sol at medium and max effort and by 26.3 and 26.7 points for GPT-5.6 Terra.”

In the paper

“Among the remaining mechanisms, diagnosis feedback matters most, followed by execution feedback.”

In the paper

“Inter-primitive communication matters more on SWE Refactor Bench and Terminal-Bench, 7.5 and 5.7 points, than on SWE-bench Verified, 2.8 points”

In the paper

“Removing on-demand activation costs at most 3.0 points but raises inference cost by 1.61-1.88 × .”

In the paper

“Matching the inference budget improves the GPT-5.6 Sol Editor from 38.8% to 45.5%, recovering 6.7 points, but it remains 6.3 points below HERMES at essentially the same token and monetary budget.”

In the paper

“Increasing B from 0 to 3 adds 12.5 points on SWE Refactor Bench, 12.1 on Terminal-Bench, 9.50 on DevOps-Gym, and 4.8 on SWE-bench Verified, whereas increasing B from 3 to 5 adds only 0.5, 0.9, 0.47, and 0.2 points respectively.”

In the paper

“Disabling on-demand activation also increases inference cost by 1.61-1.88 × while reducing task performance in Table 9.”

In the paper

“The centralized Editor makes fewer model calls because it processes all selected components jointly, using 10-14% fewer tokens and 11-16% lower cost than HERMES, but this reduction is accompanied by considerably lower task performance.”

In the paper

“The selected repository-visible regression suite runs 407 tests and returns the same successful exit status as before modification, providing evidence that the patch does not introduce an observed regression in this subsystem.”

In the paper

“The diagnosis module returns PASS after the first round, so no revision is triggered.”

In the paper

“Its gains are also smaller in already saturated settings, such as SWE-bench Verified with the strongest backbones.”

In the paper

“After HERMES terminates, the SWE-bench evaluator executes the held-out evaluation suite. All 35 tests pass, including the task-specific test json display for field , and the instance is marked resolved=True .”

In the paper

“Three automatically constructed reproduction attempts terminate during Django configuration before reaching the reported display behavior and are therefore discarded.”

In the paper

“The six activated Dev-Primitives then exchange 15 directed natural-language messages.”

In the paper

“The final diff therefore modifies four of the six activated components. The other two remain unchanged after inspection and communication.”

In the paper

“Interestingly, the accepted HERMES patch differs from the developer repair path. The developer patch modifies django/contrib/admin/utils.py and django/forms/fields.py , whereas HERMES does not activate django/contrib/admin/utils.py . Instead, it propagates the same Unicode-preserving constraint through lower-level JSONField and serialization paths.”

In the paper

“The activation module first identifies six repository components whose behavior may contribute to the issue and assigns each an expected action according to the schema in Appendix C. As shown in Table 16, four components are assigned modify , while two are assigned inspect .”

In the paper

“However, existing agents remain brittle on long-horizon workflows, where they must repeatedly reconstruct program state scattered across source files, configurations, tests, dependencies, and runtime behavior, leading to increasingly long interaction histories, context explosion, and semantic drift.”

• Also stated in: § 1 INTRODUCTION · p. 1 · plain

  “Both paradigms remain brittle once such interactions extend over long horizons. Append-only histories and passive compression lead to context explosion, semantic drift, and degraded reasoning (Liu et al., 2026; Wang et al., 2026a).”

What the sources say

• Not checked yetcites Liu et al., 2026
• Not checked yetcites Wang et al., 2026a

In the paper

“context-management methods keep information inside a single trajectory (Liu et al., 2026; Wang et al., 2026a)”

What the sources say

• Not checked yetcites Liu et al., 2026
• Not checked yetcites Wang et al., 2026a

In the paper

“so that component-specific information and reasoning stay with the artifact that owns them.”

In the paper

“prior primitives encapsulate reusable, task-agnostic behaviors (Jin et al., 2026a;b; Li et al., 2026)”

What the sources say

• Not checked yetcites Jin et al., 2026a
• Not checked yetcites Jin et al., 2026b
• Not checked yetcites Li et al., 2026

In the paper

“Unlike conventional agents that repeatedly retrieve repository files as passive context, Dev-Primitives turn these components into active participants that can interpret natural-language requests, communicate implementation requirements with other components, and directly edit their own artifacts.”

In the paper

“The activated primitives modify their own artifacts and are validated in an execution environment.”

In the paper

“We therefore introduce HERMES , a H arness E ngineering framework for software enginee R ing via M odular E xecutable Dev-Primitive S , which adds two mechanisms.”

In the paper

“Each primitive has direct access to the artifact it represents and is responsible for reasoning about and modifying that artifact during software engineering.”

In the paper

“The message m i carries what the primitive discovered while reasoning over its own implementation and what it therefore requires of other components: interface changes, implementation requirements, dependency updates, configuration constraints, or testing requirements.”

In the paper

“Through this interface it interprets requests, reasons over its own artifact, communicates requirements and constraints to other primitives, and modifies its implementation when necessary, so that component-specific information and reasoning stay with the artifact that owns them.”

In the paper

“A Dev-Primitive is instead defined by the artifact it owns, so the set of primitives is induced by the repository and requirements flow along its dependency edges”

In the paper

“we introduce Dev-Primitives (Development Primitives), a modular and executable abstraction that transforms repository components from passive software artifacts into active participants in software engineering.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “We answer this question with Dev-Primitives (Development Primitives), modular and executable abstractions that transform repository components from passive code artifacts into active participants.”

In the paper

“Dynamic activation localizes the components likely to implement the reported behavior, follows their dependencies to the callers, configurations, and tests that may need to change with them, and instantiates only the Dev-Primitives in this candidate set, which modify their artifacts and exchange requirements before the repository is executed.”

In the paper

“Because a single round often leaves the issue unresolved, and execution reports only that the repository still fails rather than which component should be revised, a bug diagnosis mechanism traces the failure back to the components it implicates and reactivates only those primitives, repeating until the repository passes or a revision budget is exhausted.”

In the paper

“A Dev-Primitive therefore performs local modification, reasoning over and editing the artifact it represents, and inter-primitive communication, exchanging task-relevant information with other Dev-Primitives through natural language.”

In the paper

“A Dev-Primitive wraps an individual repository component, such as a source file, configuration file, or test file, with a resident LLM.”

In the paper

“HERMES activates these primitives on demand and revises them from execution evidence.”

In the paper

“Each Dev-Primitive pairs a repository artifact with a resident LLM, which gives the artifact an agent-native interface grounded in its own implementation and dependencies, enabling natural-language reasoning, inter-component communication, and localized self-modification.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “Each Dev-Primitive wraps a repository artifact with a resident LLM, and this is what makes the artifact agent-native: because the resident model reads the implementation it is attached to, the component can be addressed in natural language and can answer for its own implementation and dependencies.”

In the paper

“HERMES revises the plan as Π ′ = ACTIVATE( q, R ′ , Π , ϕ ) , which may retune objectives, drop components, or activate ones the new evidence implicates, and the loop continues until the repository state is accepted or a budget of B rounds is exhausted.”

In the paper

“The component set named in c is what keeps revision localized.”

In the paper

“Participation does not imply modification: a primitive may contribute information about its implementation while leaving its own artifact unchanged.”

In the paper

“We follow the official leaderboard settings and evaluation protocols of each benchmark”

In the paper

“Rather than relying on model-side reasoning about correctness, HERMES evaluates the modified repository R ′ in an executable environment, producing o = EXECUTE( R ′ ) : command outputs and exit codes, test outcomes, runtime errors, and available logs and stack traces.”

In the paper

“Repeating the default GPT-5.6 Sol configuration three times yields 31 . 0 ± 1 . 3 (Appendix G), showing that the gains are not due to run-to-run variation.”

In the paper

“In every result table, each group compares a baseline harness with HERMES on the identical backbone and reasoning effort; only the harness differs”

In the paper

“Messages are directed rather than broadcast.”

In the paper

“Bug diagnosis supplies this attribution, returning v = DIAGNOSE( q, Π , R ′ , o ) ∈ { PASS , FAIL } and, when v = FAIL, structured feedback ϕ = ( e, c, u ) : the observed failure, the suspected root cause together with the components implicated by the evidence, and revision guidance.”

In the paper

“Note that held-out evaluation tests are never used during solving (Appendix F.1).”

In the paper

“It localizes the components likely to implement the reported behavior and expands along their dependencies to callers, callees, configurations, and tests, inspecting repository structure and retrieving files as needed rather than loading the repository into one context.”

In the paper

“Each receives its local objective x i from the plan together with the messages C i addressed to it by other activated primitives, reasons over its own artifact in light of both, and produces ( a ′ i , m i ) = P i ( a i , x i , C i ) : an optionally modified artifact and an outgoing message carrying the requirements that its modification imposes elsewhere.”

In the paper

“Messages produced in a round enter the communication context C j of their recipients before that round terminates, so a primitive may revise its own modification after receiving a requirement from another component.”

In the paper

“A primitive sends only to the components its local reasoning implicates, which are in practice its callers, callees, configurations, and tests, so the communication topology follows the repository's dependency structure rather than a predefined organization.”

In the paper

“Messages circulate only among the primitives in P A and reach their recipients before the round terminates, so a requirement discovered while editing one component can still be satisfied by the components that depend on it within the same round.”

In the paper

“Table 8 shows that dynamic activation recovers most target components while activating only a small subset of repository components.”

In the paper

“Neither mechanism can override deterministic execution outcomes: a trajectory with failing task-visible tests, unsuccessful compilation, or other observed execution failures cannot be accepted on the basis of model judgment alone.”

In the paper

“a monolithic agent has no such routing problem because it has no owners.”

In the paper

“once edits are artifact-local, a failure observed at the repository level must be routed back to the component that owns the responsible implementation”

In the paper

“Held-out evaluation tests are not exposed during solving (Appendix F.1).”

In the paper

“HERMES evaluates the modified repository through actual execution using the repository snapshots, dependencies, and runtime environments provided by each benchmark.”

In the paper

“We follow the same separation for SWE Refactor Bench, Terminal-Bench 4.0, and DevOps-Gym: benchmark-specific graders and held-out evaluation checks are reserved for final scoring rather than being returned as feedback during solving.”

In the paper

“During a trajectory, dynamic activation, the Dev-Primitives, and bug diagnosis may use only information available from the task repository and its execution environment, including repository-provided tests, build commands, linters, type checkers, runtime outputs, stack traces, logs, and reproduction scripts constructed during solving.”

In the paper

“Dev-Primitives instead pair each repository component with a resident LLM that reasons over the component's current implementation, functionality, and dependencies.”

In the paper

“Building on this artifact-bound representation, HERMES dynamically activates task-relevant Dev-Primitives and coordinates their modification through environment execution, diagnosis feedback, and revision.”

In the paper

“Second, existing primitive- and skill-based methods encapsulate reusable, task-agnostic behaviors, reasoning procedures, or tools (Jin et al., 2026a;b; Li et al., 2026), whereas a Dev-Primitive is bound to a specific mutable artifact and evolves together with that artifact throughout execution.”

What the sources say

• Not checked yetcites Jin et al., 2026a;b; Li et al., 2026

In the paper

“Reference-patch overlap is nonetheless a conservative measure of localization”

In the paper

“The diagnosis module cannot override deterministic execution failures.”

In the paper

“A trajectory with failing task-visible tests, unsuccessful compilation, or other observed execution failures cannot be marked as PASS solely based on model judgment.”

In the paper

“Benchmark-held-out evaluation tests and final grading outcomes are not exposed to HERMES during task execution.”

In the paper

“The execution environment does not rely on an LLM to predict whether a modification is correct.”

In the paper

“In particular, Tool Primitives expose tools through agent-native natural-language interfaces, hiding schema resolution and execution details from the calling model.”

In the paper

“SWE-Gym (Pan et al., 2024), SWEsmith (Yang et al., 2026), and Lingma SWE-GPT (Ma et al., 2024) learn software engineering behavior from repository-level trajectories”

What the sources say

• Not checked yetcites Pan et al., 2024
• Not checked yetcites Yang et al., 2026
• Not checked yetcites Ma et al., 2024

In the paper

“Dev-Primitives adopt a related modular perspective, but attach the capability to mutable repository artifacts rather than reusable task-agnostic behaviors or external tools.”

In the paper

“Even when individual developers are responsible for specific files, coordination is organized through a manager, shared plan, or design contract, so both the set of reasoning units and the paths along which they exchange information are fixed by the designer rather than by the repository.”

In the paper

“These methods modularize reusable reasoning procedures, skills, or tool-use capabilities so that they can be invoked and composed by an agent.”

In the paper

“Third, context-management approaches primarily compress, retrieve, or reorganize information within a centralized agent trajectory (Liu et al., 2026; Wang et al., 2026a).”

What the sources say

• Not checked yetcites Liu et al., 2026
• Not checked yetcites Wang et al., 2026a

In the paper

“Other work represents reusable capabilities explicitly through modular abstractions, including Agent Primitives (Jin et al., 2026a), Tool Primitives (Jin et al., 2026b), AgentDistill (Qiu et al., 2025), and CODESKILL (Li et al., 2026).”

What the sources say

• Not checked yetcites Jin et al., 2026a
• Not checked yetcites Jin et al., 2026b
• Not checked yetcites Qiu et al., 2025

In the paper

“Additional inference therefore accounts for part, but not all, of the gap.”

In the paper

“P2T (Ma et al., 2026) emphasizes high-quality process supervision.”

What the sources say

• Not checked yetcites Ma et al., 2026

In the paper

“The issue reports that non-ASCII characters stored in a JSONField are displayed in the Django admin as escaped Unicode sequences rather than as the original characters.”

In the paper

“Dynamic activation thus limits both unnecessary model calls and interference from components that are no longer required at the current stage of the task.”

In the paper

“Finally, Qwen3-8B is served locally with Ollama on NVIDIA A40 GPUs using the model's default sampling parameters (temperature 0.6, topp 0.95, topk 20) and Ollama's 32K context window, while all other models are accessed through their official APIs; results may therefore vary with deployment settings and model updates.”

In the paper

“In addition, performance depends on correctly activating task-relevant components and on informative execution feedback for revision; missed dependencies or sparse runtime signals can therefore limit recovery.”