An Empirical Study of Harness Design for Coding Agents

Classified as probably not a substantial claim.

The harness, not the model alone, governs performance.

“Research on their design has shown that the harness, not the model alone, governs performance (Yang et al., 2024).”

• cites Yang et al., 2024 sourceSupported

May combine two claims.

Making elided content recoverable adds machinery that models rarely use and yields no accuracy gain.

“whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain.”

• Also stated in: § 1 Introduction · p. 3 · plain

  “In contrast, the recall mechanism that makes elision reversible is rarely invoked and does not improve accuracy over elision alone.”

May combine two claims.

T4 achieves success rates comparable to T1-T3, with the lowest cost in seven of eight model-benchmark panels.

“Figure 4 shows that T4 achieves success rates comparable to T1-T3, with the lowest cost in seven of eight model-benchmark panels.”

May combine two claims.

For stronger models, planning mainly removes redundant post-edit verification, lowering cost with only small changes in accuracy.

“for the stronger models, it mainly removes redundant post-edit verification, lowering cost with only small changes in accuracy.”

May combine two claims.

Bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks.

“whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks.”

• Also stated in: § 1 Introduction · p. 3 · plain

  “Predefined tools reduce reliance on shell commands, whereas bash-capable models can combine multiple operations per call.”

May combine two claims.

At 32k, T1 and T2 trajectories still reach approximately the full window, whereas T3 and T4 keep peak context substantially below it.

“Figure 5(a) shows that at 32k, T1 and T2 trajectories still reach approximately the full window, whereas T3 and T4 keep peak context substantially below it.”

May combine two claims.

The model-averaged T0 overflow rate falls from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench across 32k to 128k budgets.

“Across these budgets, the model-averaged T0 overflow rate falls from 78 . 7% to 8 . 7% on SWE-Bench and from 61 . 0% to 12 . 1% on Terminal-Bench, while all managed tiers have zero overflow failures throughout.”

May combine two claims.

Unlike 30B, 120B does not benefit from longer trajectories: the predefined tool set shortens average runs from 101 to 77 turns on SWE-Bench and 96 to 70 on Terminal-Bench without increasing cost.

“Unlike 30B, 120B does not benefit from longer trajectories: the predefined tool set shortens average runs from 101 to 77 turns on SWE-Bench and 96 to 70 on Terminal-Bench without increasing cost.”

May combine two claims.

Planning improves success rate for the weaker model and saves cost for the stronger models.

“Planning improves success rate for the weaker model and saves cost for the stronger models.”

• Also stated in: § 6 Conclusion · p. 15 · plain

  “Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.”

May combine two claims.

For Nemotron-3 120B, trajectory-length distributions nearly overlap, consistent with the absence of an accuracy gain.

“For Nemotron-3 120B, trajectory-length distributions nearly overlap, consistent with the absence of an accuracy gain.”

May combine two claims.

For Nemotron-3 30B, planning increases success rate by 11.6 percentage points on SWE-Bench and 4.5 points on Terminal-Bench, at higher cost on both benchmarks.

“For Nemotron-3 30B, planning increases success rate by 11 . 6 percentage points on SWE-Bench and 4 . 5 points on Terminal-Bench, at higher cost on both benchmarks.”

May combine two claims.

32.8% of Mistral's bash-only SWE-Bench runs end without editing a file, versus 1.2% with the full tool set.

“Additional bash-only failures arise before repair: 32.8% of Mistral's bash-only SWE-Bench runs end without editing a file, versus 1.2% with the full tool set (Table 12), and among unresolved runs, the share never reaching the correct file rises from 16.0”

• cites Table 12Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Without planning, 68.6% of Nemotron-3 30B runs terminate without an edit and 58.4% terminate during Localize, compared with 27.8% and 10.4%, respectively, with planning.

“Without planning, 68.6% of runs terminate without an edit and 58.4% terminate during Localize, compared with 27.8% and 10.4%, respectively, with planning (Table 6).”

• cites Table 6Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

On SWE-Bench, Nemotron-3 550B and Mistral-Medium-3.5-128B costs fall by approximately 30% and 32%, respectively, while success rates decrease by 2.0 and 0.4 percentage points.

“On SWE-Bench, their costs fall by approximately 30% and 32% , respectively, while success rates decrease by 2 . 0 and 0 . 4 percentage points (Tables 3 and 4).”

• cites Tables 3 and 4Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Recall (M2) is rarely used and does not improve accuracy over elision alone.

“Recall (M2) is rarely used and does not improve accuracy over elision alone.”

• Also stated in: § 3.2 Main Results · p. 8 · hedged

  “These results suggest that lossless storage adds machinery most models seldom use, and retrieving elided observations does not consistently translate into completed tasks.”

May combine two claims.

Without context management, the proportion of active SWE-Bench runs declines sharply, with median lengths of 20-30 turns across the four models.

“Without context management, the proportion of active SWE-Bench runs declines sharply, with median lengths of 20-30 turns across the four models.”

May combine two claims.

The effect is smaller for Nemotron-3 550B (3.6%), whose exploratory tail is less compressed, but larger for Mistral-Medium-3.5-128B (about 40%), where both medium- and long-running trajectories shorten.

“The effect is smaller for Nemotron-3 550B (3.6%), whose exploratory tail is less compressed, but larger for Mistral-Medium-3.5-128B (about 40%), where both medium- and long-running trajectories shorten.”

May combine two claims.

The full tool set improves Mistral-Medium-3.5-128B success by 23.2% on SWE-Bench, but bash-only adds 6.7% on Terminal-Bench.

“The full tool set improves success by 23 . 2% on SWE-Bench, but bash-only adds 6 . 7% on Terminal-Bench.”

May combine two claims.

With all tools, Mistral issues 71.9% of Terminal-Bench workspace actions via bash versus 40.4% on SWE-Bench, while predefined-tool use falls from 31.4 to 13.1 calls per task.

“Figure 7(c) explains the reversal: with all tools, Mistral issues 71 . 9% of Terminal-Bench workspace actions via bash versus 40 . 4% on SWE-Bench, while predefined-tool use falls from 31 . 4 to 13 . 1 calls per task.”

May combine two claims.

Most runs terminate during the Localize phase, and few reach Verify.

“Most runs terminate during the Localize phase, and few reach Verify (Figure 32).”

• cites Figure 32Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Bash-only trajectories issue 32% fewer calls on SWE-Bench and 24% fewer on Terminal-Bench.

“Bash-only trajectories issue 32% fewer calls on SWE-Bench and 24% fewer on Terminal-Bench.”

May combine two claims.

The predefined tool set scaffolds weaker models, while bash-only can benefit stronger ones.

“The predefined tool set scaffolds weaker models, while bash-only can benefit stronger ones.”

• Also stated in: § 6 Conclusion · p. 15 · boosted

  “Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.”

May combine two claims.

For Nemotron-3 550B on Terminal-Bench, switching from the full tool set to bash-only shortens the median trajectory from 47 to 31 actions while raising the share of Write-code actions from 16% to 27%.

“Figure 9 shows this most clearly for Nemotron-3 550B on Terminal-Bench: switching from the full tool set to bash-only shortens the median trajectory from 47 to 31 actions while raising the share of Write-code actions from 16% to 27%, so a larger fraction of a shorter trajectory is spent producing code.”

May combine two claims.

Planning reduces the median trajectory length from 108 to 74 turns for Nemotron-3 550B and from 68 to 53 turns for Mistral-Medium-3.5-128B.

“Figure 8 shows that planning reduces the median trajectory length from 108 to 74 turns for Nemotron-3 550B and from 68 to 53 turns for Mistral-Medium-3.5-128B.”

May combine two claims.

For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning reduces cost on both benchmarks, accompanied by small decreases in success rate.

“For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning reduces cost on both benchmarks, accompanied by small decreases in success rate.”

May combine two claims.

For Nemotron-3 120B, accuracy gains shrink to 1.6% on SWE-Bench and 4.5% on Terminal-Bench but accompany more efficient execution.

“For Nemotron-3 120B, accuracy gains shrink to 1 . 6% on SWE-Bench and 4 . 5% on Terminal-Bench but accompany more efficient execution.”

May combine two claims.

Median trajectory lengths range from 39-42 turns for Nemotron-3 30B and 70-74 turns for Nemotron-3 550B.

“Median trajectory lengths range from 39-42 turns for Nemotron-3 30B and 70-74 turns for Nemotron-3 550B.”

May combine two claims.

For Nemotron-3 30B, the predefined tool set raises success by 15.0% on SWE-Bench and 10.1% on Terminal-Bench.

“For Nemotron-3 30B, the predefined tool set raises success by 15.0% on SWE-Bench and 10.1% on Terminal-Bench.”

May combine two claims.

For Nemotron-3 550B, bash-only improves success rate by 3.6% on SWE-Bench and 5.6% on Terminal-Bench while reducing cost by 53% and 30%, respectively.

“For Nemotron-3 550B, bash-only improves success rate by 3 . 6% on SWE-Bench and 5 . 6% on Terminal-Bench while reducing cost by 53% and 30% , respectively.”

May combine two claims.

Planning increases turns, tool calls, and average input tokens per call for Nemotron-3 30B on both benchmarks.

“Planning increases turns, tool calls, and average input tokens per call for Nemotron-3 30B on both benchmarks.”

May combine two claims.

Planning has two opposing effects: it extends trajectories that would otherwise terminate prematurely, while truncating the tail of excessively long runs.

“Figure 9 shows that planning has two opposing effects: it extends trajectories that would otherwise terminate prematurely, while truncating the tail of excessively long runs.”

May combine two claims.

Even the heaviest-use configuration, Nemotron-3 30B on Terminal-Bench at 32k under T2, averages 4.326 calls per task and scores 3.37% below T1.

“Even the heaviest-use configuration, Nemotron-3 30B on Terminal-Bench at 32k under T2, averages 4 . 326 calls per task and scores 3 . 37% below T1.”

May combine two claims.

All managed tiers extend execution while largely preserving phase ordering and proportions at comparable turns.

“All managed tiers extend execution while largely preserving phase ordering and proportions at comparable turns.”

May combine two claims.

For Nemotron-3 120B, planning truncates the long-running tail and reduces cost by roughly 26%.

“For Nemotron-3 120B, planning truncates the long-running tail and reduces cost by roughly 26%.”

May combine two claims.

Median lengths increase to approximately 50-180 turns, depending on the model and tier, allowing runs to progress to verification.

“Median lengths increase to approximately 50-180 turns, depending on the model and tier, allowing runs to progress to verification.”

May combine two claims.

Planning produces little change in the fraction of runs terminating without an edit or failing at file localization, while substantially reducing turns and tool calls.

“Consistently, planning produces little change in the fraction of runs terminating without an edit or failing at file localization (see Tables 12 and 10), while substantially reducing turns and tool calls (see Table 11).”

• cites Tables 12 and 10Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Table 11Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For Nemotron-3 120B, planning yields no consistent success-rate gain; it increases cost on SWE-Bench but reduces it on Terminal-Bench.

“For Nemotron-3 120B, planning yields no consistent success-rate gain; it increases cost on SWE-Bench but reduces it on Terminal-Bench.”

May combine two claims.

Re-patch counts differ by fewer than two per task, while the proportion of runs terminating without an edit varies by less than four percentage points for three of the four models.

“Re-patch counts differ by fewer than two per task, while the proportion of runs terminating without an edit varies by less than four percentage points for three of the four models (Figures 35 and 39; Tables 11 and 12).”

• cites Figures 35 and 39Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Tables 11 and 12Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

All three measures decrease for Mistral on both benchmarks and for Nemotron-3 550B on SWE-Bench; the changes for Nemotron-3 550B on Terminal-Bench are smaller.

“All three measures decrease for Mistral on both benchmarks and for Nemotron-3 550B on SWE-Bench; the changes for Nemotron-3 550B on Terminal-Bench are smaller.”

May combine two claims.

On Terminal-Bench, 66% of bash-only trajectories terminate after such out-of-interface emissions, shortening the average trajectory from 71 to 15 turns.

“On Terminal-Bench, 66% of bash-only trajectories terminate after such out-of-interface emissions, shortening the average trajectory from 71 to 15 turns.”

May combine two claims.

Across 32 model-benchmark-window comparisons, T2 outperforms T1 in 15 settings, underperforms in 14, and ties in three; the equal-weight mean difference is -0.36 percentage points (+0.40 on SWE-Bench, -1.12 on Terminal-Bench).

“Across 32 modelbenchmark-window comparisons, T2 outperforms T1 in 15 settings, underperforms in 14, and ties in three; the equal-weight mean difference is -0 . 36 percentage points ( +0 . 40 on SWE-Bench, -1 . 12 on Terminal-Bench).”

May combine two claims.

For Terminal-Bench action labels, raw agreement ranges from 79.8% to 97.3% and κ ranges from 0.758 to 0.965.

“Terminal-Bench action labels also show substantial agreement, with raw agreement ranging from 79.8% to 97.3% and κ ranging from 0.758 to 0.965.”

May combine two claims.

For SWE-Bench action labels, raw agreement ranges from 92.7% to 99.0% and κ ranges from 0.881 to 0.984.

“Agreement is especially strong for SWE-Bench action labels, where raw agreement ranges from 92.7% to 99.0% and κ ranges from 0.881 to 0.984.”

May combine two claims.

Across T0-T4, re-patch counts vary by less than two per task and median edit sizes by a few lines.

“while re-patch counts vary by less than two per task and median edit sizes by a few lines.”

May combine two claims.

Across T0-T4, the median trajectory length and the mean number of tool calls vary only within a narrow band for every model.

“Across T0-T4, the median trajectory length and the mean number of tool calls vary only within a narrow band for every model (Table 11)”

Classified as probably not a substantial claim. May combine two claims.

The validation covers 15,610 labeled units in total, including 5,306 Terminal-Bench action labels, 10,254 SWE-Bench action labels, and 50 SWE-Bench failure-stage diagnoses.

“The validation covers 15,610 labeled units in total, including 5,306 Terminal-Bench action labels, 10,254 SWE-Bench action labels, and 50 SWE-Bench failure-stage diagnoses.”

May combine two claims.

Aggregating across the three splits gives approximately 94.2% raw agreement and a weighted mean Cohen's κ of 0.929.

“Aggregating across the three splits gives approximately 94.2% raw agreement and a weighted mean Cohen's κ of 0.929.”

May combine two claims.

Across all four models, bash-only reduces repeated patching of already edited files, from 3.3 to 0.4 re-patches for Nemotron-3 30B, 2.8 to 2.2 for 120B, 4.6 to 1.5 for 550B, and 3.0 to 1.3 for Mistral.

“Across all four models, bash-only reduces repeated patching of already edited files, from 3.3 to 0.4 re-patches for Nemotron-3 30B, 2.8 to 2.2 for 120B, 4.6 to 1.5 for 550B, and 3.0 to 1.3 for Mistral.”

May combine two claims.

For Mistral-Medium-3.5-128B, removing planning lengthens the trajectory from 53 to 68 median turns while leaving the without-edit rate below 3%.

“whereas for Nemotron-3 550B and Mistral-Medium-3.5-128B it lengthens the trajectory (from 74 to 108 and from 53 to 68 median turns) while leaving the without-edit rate below 3%”

May combine two claims.

Termination behavior is similarly stable: the fraction of SWE-Bench runs that terminate without an edit varies across tiers by less than four points for three of the four models and by about eight points for Nemotron-3 120B.

“Termination behavior is similarly stable (Table 12): the fraction of SWE-Bench runs that terminate without an edit varies across tiers by less than four points for three of the four models and by about eight points for Nemotron-3 120B.”

May combine two claims.

The per-split overall agreement ranges from 88.7% to 98.4%, with Cohen's κ ranging from 0.858 to 0.980.

“The per-split overall agreement ranges from 88.7% to 98.4%, with Cohen's κ ranging from 0.858 to 0.980.”

May combine two claims.

For Nemotron-3 550B, removing planning lengthens the trajectory from 74 to 108 median turns while leaving the without-edit rate below 3%.

“whereas for Nemotron-3 550B and Mistral-Medium-3.5-128B it lengthens the trajectory (from 74 to 108 and from 53 to 68 median turns) while leaving the without-edit rate below 3%”

May combine two claims.

For failure-stage diagnosis, agreement ranges from 87.5% to 100.0%, with κ ranging from 0.813 to 1.000.

“For failure-stage diagnosis, agreement remains high, ranging from 87.5% to 100.0%, with κ ranging from 0.813 to 1.000.”

May combine two claims.

As more unmanaged trajectories fit within the window, context management's marginal accuracy benefit shrinks and becomes more model-dependent.

“as more unmanaged trajectories fit within the window, its marginal accuracy benefit shrinks and becomes more model-dependent.”

May combine two claims.

Comparisons between complete harnesses conflate multiple mechanisms, so a performance difference between two agents does not reveal whether the gain comes from planning, tool design, context management, or their interaction with the underlying model.

“However, comparisons between complete harnesses conflate multiple mechanisms, so a performance difference between two agents does not reveal whether the gain comes from planning, tool design, context management, or their interaction with the underlying model.”

Classified as probably not a substantial claim.

The comparison between predefined tools and bash-only should be interpreted as the effect of the complete action interface, including tool availability, interface instructions, state tracking, and validation support, rather than as the isolated effect of tool count or action granularity.

“The comparison should therefore be interpreted as the effect of the complete action interface, including tool availability, interface instructions, state tracking, and validation support, rather than as the isolated effect of tool count or action granularity.”

May combine two claims.

The predefined tool set scaffolds weaker models that cannot reliably express intent through bash alone, but can add overhead for stronger models capable of composing complex, multi-step shell commands; the crossover also depends on how shell-centric the task types is.

“Takeaway: The predefined tool set scaffolds weaker models that cannot reliably express intent through bash alone, but can add overhead for stronger models capable of composing complex, multi-step shell commands. The crossover also depends on how shell-centric the task types is.”

May combine two claims.

Planning helps the least capable model sustain execution through an initial edit attempt, with the additional turns contributing to higher cost.

“These results suggest that planning helps the least capable model sustain execution through an initial edit attempt, with the additional turns contributing to higher cost.”

May combine two claims.

The Mistral result is an accuracy-cost trade-off rather than uniform dominance, showing that the preferred action space depends on both capability and task types.

“The Mistral result is therefore an accuracy-cost trade-off rather than uniform dominance, showing that the preferred action space depends on both capability and task types.”

May combine two claims.

Bash-only enables larger code-writing actions and fewer interactions.

“Bash-only enables larger code-writing actions and fewer interactions.”

May combine two claims.

Planning trades additional computation for accuracy on the weaker model, but primarily reduces cost on the stronger models; its value at intermediate capability remains task-type-dependent.

“Takeaway: Planning trades additional computation for accuracy on the weaker model, but primarily reduces cost on the stronger models; its value at intermediate capability remains task-type-dependent.”

May combine two claims.

T4's early elision handles many cases before summarization is needed, reducing costly LLM summarization calls and helping explain its lower cost.

“These results suggest that T4's early elision handles many cases before summarization is needed, reducing costly LLM summarization calls and helping explain its lower cost.”

Classified as probably not a substantial claim.

Other supporting components (workspace access controls, post-edit diagnostics, stuck detection) remain fixed across ablations to isolate each intervention's effect.

“Other supporting components like, workspace access controls, post-edit diagnostics, and stuck detection remain fixed across ablations to isolate each intervention's effect.”

Classified as probably not a substantial claim.

Harness preferences can vary across models.

“suggesting that harness preferences can vary across models.”

• cites Cao et al., 2026 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Success rates were compared using two-sided exact McNemar tests on task-paired outcomes, with Benjamini-Hochberg procedure to control the false discovery rate at 0.05.

“Within each family, we compare success rates using two-sided exact McNemar tests on task-paired outcomes and apply the Benjamini-Hochberg procedure to control the false discovery rate at 0.05.”

Classified as probably not a substantial claim. May combine two claims.

Elision reclaims tokens cheaply but discards detail, recall recovers that detail when needed, and summarization compresses history too old to keep verbatim.

“Elision reclaims tokens cheaply but discards detail, recall recovers that detail when needed, and summarization compresses history too old to keep verbatim.”

May combine two claims.

Planning extends Nemotron-3 30B runs that would otherwise terminate before an edit, while the removed turns for Nemotron-3 550B and Mistral-Medium-3.5-128B are largely post-edit verification.

“Section 4 traces these changes to individual trajectories: planning extends Nemotron-3 30B runs that would otherwise terminate before an edit, while the removed turns for Nemotron-3 550B and Mistral-Medium-3.5-128B are largely post-edit verification.”

May combine two claims.

The shell lets capable models bundle several low-level operations into one command or script, whereas the structured interface spreads the same work across many smaller interactions.

“This pattern suggests that the shell lets capable models bundle several low-level operations into one command or script, whereas the structured interface spreads the same work across many smaller interactions.”

May combine two claims.

Recall use is concentrated under the greatest context pressure and almost entirely in Nemotron-3 30B; Nemotron-3 550B and Mistral-Medium-3.5-128B rarely invoke it.

“Recall use is therefore concentrated under the greatest context pressure and almost entirely in Nemotron-3 30B; Nemotron-3 550B and Mistral-Medium-3.5-128B rarely invoke it.”

May combine two claims.

Terminal-Bench is more shell-centric, so removing competing workspace tools better matches the model's preferred actions; on SWE-Bench, predefined read, search, and edit actions remain important.

“Terminal-Bench is therefore more shell-centric, so removing competing workspace tools better matches the model's preferred actions; on SWE-Bench, predefined read, search, and edit actions remain important.”

May combine two claims.

The harness's modular design allows components to be independently configured and composed, enabling controlled component-level analysis.

“Our modular design allows components to be independently configured and composed, enabling controlled component-level analysis.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “This modular design allows us to hold the model, task, execution loop, and unablated components fixed while estimating the conditional effect of each implemented intervention.”

Classified as probably not a substantial claim.

Web search was excluded because SWE-Bench tasks originate from public GitHub issues, and search could expose the corresponding pull request and ground-truth patch.

“We exclude web search because SWE-Bench tasks originate from public GitHub issues, and search could expose the corresponding pull request and ground-truth patch (Cao et al., 2026).”

• cites Cao et al., 2026 sourceNot checked — no full text of the source

May combine two claims.

Planning and the action space are the interventions that change the trajectory.

“Planning and the action space are the interventions that change the trajectory.”

May combine two claims.

Predefined tools lower the complexity of each individual action, but they often do so by increasing the number of interactions and incremental repair cycles needed to express the same operation.

“Thus, predefined tools lower the complexity of each individual action, but they often do so by increasing the number of interactions and incremental repair cycles needed to express the same operation.”

May combine two claims.

Each setting is run once per task, and Terminal-Bench contains only 89 tasks, so many Terminal-Bench contrasts do not reach significance under the paired McNemar test.

“Each setting is run once per task, and Terminal-Bench contains only 89 tasks, so many Terminal-Bench contrasts do not reach significance under the paired McNemar test; our conclusions there rest on consistent directions across models and budgets rather than on individually significant cells.”

Classified as probably not a substantial claim.

Behavioral differences between coding-agent frameworks change across model generations.

“AgentArch (Bogavelli et al., 2025) reports model-specific architecture preferences, while a large trajectory study finds that behavioral differences between coding-agent frameworks change across model generations (Mehtiyev and Assunção, 2026).”

• cites Bogavelli et al., 2025 sourceNot checked — no full text of the source
• cites Mehtiyev and Assunção, 2026 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

When enabled, the update_plan and recall_event tools remain available under either action-space condition.

“When enabled, both tools remain available under either action-space condition.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The SWE-Bench behavior encoding builds on the trajectory-analysis framework of Mehtiyev and Assunção (2026).

“Building on the trajectory-analysis framework of Mehtiyev and Assunção (2026), we assign every turn to one of five phases: localization, reproduction, fixing, verification, and other auxiliary behavior.”

• cites Mehtiyev and Assunção (2026) sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Individual interventions can become less useful or even harmful for stronger models, as observed for prompt-engineering strategies in software engineering.

“Individual interventions can likewise become less useful or even harmful for stronger models, as observed for prompt-engineering strategies in software engineering (Wang et al., 2026).”

• cites Wang et al., 2026 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The preferred design depends on the model.

“Finer-grained studies further indicate that the preferred design depends on the model.”

Classified as probably not a substantial claim.

Planning and the action space are ablated only under the default T4/128k configuration due to limited computing resources.

“Planning and the action space are ablated only under the default T4/128k configuration due to limited computing resources, and a full factorial study would be required to determine whether their effects persist under other combinations of components and context budgets.”

May combine two claims.

The action-space intervention (ablation) changes both the active tool registry / tool availability and the interface-specific system instructions through which the tools are presented to the model, so the ablation is not isolated to tool availability alone.

“The action-space intervention changes both the active tool registry and the interface-specific system instructions.”

• Also stated in: § A.1 System prompts · p. 19 · plain

  “Consequently, the action-space ablation changes both tool availability and the instructions through which those tools are presented to the model.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Using the earliest-failure convention for assigning failure stages to unresolved SWE-Bench trajectories ensures that downstream symptoms, such as failed tests, do not obscure earlier localization or implementation errors.

“Using the earliest-failure convention ensures that downstream symptoms, such as failed tests, do not obscure earlier localization or implementation errors.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Permission denials do not count as failures for the purpose of stuck detection.

“Permission denials do not count as failures for this purpose.”

Classified as probably not a substantial claim.

For each split and unit type, annotators independently assign labels using the same inventories provided to the LLM judge.

“For each split and unit type, annotators independently assign labels using the same inventories provided to the LLM judge.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Planning is instantiated through one prompt and update mechanism, and context management follows one threshold-based policy with fixed compaction settings.

“Planning is instantiated through one prompt and update mechanism, and context management follows one threshold-based policy with fixed compaction settings.”

May combine two claims.

The value of managing context cannot be separated from how much window there is to manage against, nor from whether the model is strong enough to be helped rather than starved by compaction.

“The value of managing context cannot be separated from how much window there is to manage against, nor from whether the model is strong enough to be helped rather than starved by compaction.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The validation sample consisted of 200 trajectories divided evenly into three non-overlapping splits, one per annotator, with one annotator labeling one more trajectory than the other two.

“A sample of 200 trajectories is divided evenly into three non-overlapping splits, one per annotator, with one annotator labeling one more trajectory than the other two; each annotator labels every unit in their split.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Stuck detection is enabled identically in every experimental condition and therefore belongs to the fixed execution substrate.

“Stuck detection is enabled identically in every experimental condition and therefore belongs to the fixed execution substrate.”