CoChat
Sign Up
PreprintCited by 1Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Open access
Zhao, Xiangxin, Li, Han, Li, Shuaiting, Zhao, Tianyi, Barr, Earl T., Sarro, Federica, Ye, He·2026·arXiv
DOI 10.48550/arxiv.2607.09510arXiv 2607.09510
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper studies CLI coding-agent failures as temporal processes rather than final outcomes, using 1,794 annotated Terminal-Bench trajectories from three scaffolds and seven frontier models. It finds failures are predominantly epistemic, begin early, often remain hidden until recovery is impossible, and that both model and scaffold affect success while epistemic errors dominate across systems.

Key findings

  • Across 1,184 failed trajectories, the median decisive error occurs at step 7; the median recovery window before lock-in is 1 execution step, with 60.9% leaving at least one recovery step, 43.9% leaving three or more, and 239 trajectories remaining recoverable for over ten steps. The median first observable failure is around step 16, about ten steps after the decisive error.
  • Root causes of decisive errors are epistemic in 57.9% of cases, competence-related in 32.8%, and environment-related in 9.4%; false premises are the single largest trigger at 30.7%, followed by knowledge gaps at 24.0% and specification neglect at 14.9%.
  • Recovery differs sharply: 71% of successful trajectories recover from at least one error; successful recoveries have a median of 5 steps versus 12 for failed ones; 92% of successful trajectories respond to at least one error signal versus 37% of failed ones. Only 18% of failed trajectories terminate immediately after lock-in, while 82% continue without progress; “repairs the wrong problem” accounts for 24% of failed trajectories but 39% of wasted execution, and 26% of failed trajectories fabricate success, with 84% of fabrication beginning at or after lock-in.
  • Across the 21 model-scaffold systems, pass rates range from 19% to 45%, and epistemic errors remain the dominant failure cause in every system, accounting for 44% to 80% of failed trajectories.

Methods

  • Empirical study on Terminal-Bench: 240 tasks, three scaffolds (MiniSWE, OpenHands, Terminus2), and seven models (Claude Sonnet 4, GPT 5, Gemini 2.5 Pro, Qwen3 Coder 480B, DeepSeek V3.2, Kimi K2 Instruct, Devstral 2). From 3,843 raw trajectories, the authors retained 89 tasks completed by all 21 model-scaffold systems, yielding 1,794 valid trajectories (1,184 failed, 610 successful; over 63,000 execution steps).
  • Annotation used an LLM-assisted draft by Claude Opus 4.6, followed by two independent human annotators who finalized all labels; inter-annotator agreement was Cohen’s κ 0.78–0.94. The authors annotated three timestamps—decisive error (t_err), failure lock-in (t_lock), and first observable failure (t_obs)—plus root-cause taxonomy, post-lock-in behaviors, and recovery episodes.
  • Analyses covered RQ1 timing, RQ2 root causes, RQ3 recovery, and RQ4 model/scaffold differences. A prefix monitor (Claude Sonnet 4.6) was evaluated on 2,659 prefixes from 600 trajectories (300 failed, 300 successful) to predict whether a run had passed t_lock, using either task name alone or task name plus core requirements.

Limitations

  • Stated: construct validity relies on semantic trajectory annotation, mitigated by a fixed schema and codebook, independent annotation by two of four annotators, and high agreement (Cohen’s κ = 0.83 for root causes and weighted κ ≥ 0.94 for the three timepoints). Internal validity is observational rather than causal. External validity is limited to 1,794 trajectories covering 89 Terminal-Bench tasks and 21 model-scaffold systems; dropping timeout and incomplete trajectories may introduce bias, though the aligned set spans 16 task categories and is broadly balanced across difficulty (459 easy, 742 medium, 593 hard).
  • Evident from the text: findings are specific to Terminal-Bench, three representative scaffolds, and seven frontier models, with runs collected incrementally from May 2025 onward, so the authors describe the results as a controlled snapshot. The prefix monitor uses Claude Sonnet 4.6 and labels relative to t_lock, and the annotation pipeline uses Claude Opus 4.6 drafts even though final labels are human-determined.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.09510

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 21 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The median recovery window (t_lock - t_err) is one execution step.

    “The median recovery window (t lock -t err ) is one execution steps, with 60.9% of decisive errors leaving at least one recovery step, 43.9% leaving three or more, and 239 trajectories remaining recoverable for over ten steps.”
    Quote in the paperSubstantial claim
  • Findingown data

    Successful recoveries are much shorter than failed ones, with a median of 5 versus 12 execution steps.

    “Finding 10: Successful recoveries are much shorter than failed ones (median 5 vs. 12 execution steps).”
    Quote in the paperSubstantial claim
  • Findingown data

    Decisive errors typically begin within the first few execution steps, with a median onset of 7 execution steps.

    “typically begin within the first few execution steps”
    • Also stated in: § A. Results for RQ1 (Error Occurrence) · plain
      “Decisive errors (t err ) occur surprisingly early, with a median onset of just 7 execution steps.”
    • Also stated in: § git-workflow-hack9 • terminus2 • gpt-5 · plain
      “coding agents typically commit decisive errors early (median step 7)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.
  • Findingown data

    'Repairs the wrong problem' has the longest continuation among all post-failure behaviors, with a median of 21 remaining execution steps.

    “Although Repairs the wrong problem is not the most common post-failure behavior, it dominates wasted execution and has the longest continuation among all behaviors (median 21 steps).”
    Quote in the paperSubstantial claim
  • Findingown data

    The monitor's false-positive rate on early prefixes of successful runs stays at 2-3%.

    “This gain does not come from the monitor guessing that hardlooking tasks fail, as its false-positive rate on early prefixes of successful runs stays at 2-3%”
    Quote in the paperSubstantial claim
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Findingown data

    The median decisive error occurs at step 7, while failed runs contain a median of 27 execution steps.

    “Across all 1,184 failed trajectories, failed runs contain a median of 27 execution steps (mean 42), whereas the median decisive error occurs at only step 7 (mean 11.92).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The behavior 'Repairs the wrong problem' accounts for 24% of failed trajectories but contributes 39% of all wasted execution.

    “For example, Repairs the wrong problem accounts for only 24% of failed trajectories but contributes 39% of all wasted execution, illustrating that some recovery behaviors are disproportionately costly.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Repeatedly fixing the wrong cause accounts for 39% of all wasted execution, making it the most costly failed recovery behavior.

    “Finding 8: Repeatedly fixing the wrong cause accounts for 39% of all wasted execution, making it the most costly failed recovery behavior.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A monitor that sees only a trajectory prefix reliably recognizes failure once it has occurred, flagging locked-in runs at 82% precision, but it lacks foresight: its median lead time relative to t_lock is zero, only 3.7-8.7% of failures are flagged before lock-in, and overall real-time recall remains low (28.8% at best).

    “A monitor that sees only a trajectory prefix reliably recognizes failure once it has occurred, flagging locked-in runs at 82% precision, but it lacks foresight: its median lead time relative to t lock is zero, only 3.7-8.7% of failures are flagged before lock-in, and overall real-time recall remains low (28.8% at best).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Self-revealing failures are caught from behavior alone: environment errors reach 38% recall with only the task name and barely move (to 42%) when requirements are added.

    “Self-revealing failures announce themselves in the agent's own behavior and are caught from behavior alone: environment errors, for instance, reach 38% recall with only the task name and barely move (to 42%) when requirements are added.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among the 1,869 trajectories, 75 are unavailable due to failed executions, resulting in 1,794 valid trajectories.

    “Among these, 75 trajectories are unavailable due to failed executions, resulting in a final dataset of 1,794 valid trajectories.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    60.9% of decisive errors leave at least one recovery step, 43.9% leave three or more, and 239 trajectories remain recoverable for over ten steps.

    “The median recovery window (t lock -t err ) is one execution steps, with 60.9% of decisive errors leaving at least one recovery step, 43.9% leaving three or more, and 239 trajectories remaining recoverable for over ten steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The dataset consists of 71.4% publicly released Terminal Bench trajectories from prior work and 28.6% trajectories generated by the authors' experiments.

    “Our dataset consists of two parts: (1) publicly released Terminal Bench trajectories from prior work [29] (71.4% of our data), and (2) trajectories generated by our experiments to fill in missing scaffold and model combinations (28.6%).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Most decisive errors are epistemic rather than competence-related, with false premises being the single largest failure mode.

    “Most decisive errors are epistemic rather than competence-related, with false premises being the single largest failure mode.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The filtering yields 89 tasks, corresponding to a complete evaluation grid of 1,869 trajectories.

    “This filtering yields 89 tasks, corresponding to a complete evaluation grid of (89 × 21 = 1,869) trajectories.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The False premise trigger accounts for 30.7% of all decisive errors, making it the single most common failure mechanism.

    “This trigger alone accounts for 30.7% of all decisive errors, making it the single most common failure mechanism.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Only 18% of failed trajectories terminate shortly after the error becomes empirically unrecoverable; the remaining 82% continue executing without actual progress.

    “Finding 7: Only 18% of failed recoveries terminate immediately; the remaining 82% continue executing without actual progress.”
    • Also stated in: § 28 15 9 · plain
      “As shown in Table III, only 18% of failed trajectories terminate shortly after the error is empirically unrecoverable.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    False premises account for 30.7% of all decisive errors, making them the single most common trigger in the paper's taxonomy.

    “False premises account for 30.7% of all decisive errors, making them the single most common trigger in our taxonomy.”
    • Also stated in: § Finding 6: · plain
      “False premises are the single largest trigger of decisive errors (30.7%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Specification-relative failures can only be recognized against the requirement they violate, and the specification raises recall on ignored requirements from 3% to 22%, on false premises from 15% to 32%, and on capability gaps from 12% to 30%.

    “Specification-relative failures can only be recognized against the requirement they violate, and the specification is what unlocks them, raising recall on ignored requirements from 3% to 22%, on false premises from 15% to 32%, and on capability gaps from 12% to 30%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the representative case study, after observing the message 'sudo: not found' the agent incorrectly concluded that it lacked permission to perform the task, and this incorrect assumption became the decisive error shaping the rest of the trajectory.

    “After observing the message sudo: not found, the agent incorrectly concludes that it lacks permission to perform the task.”
    • Also stated in: § Finding 6: · plain
      “The decisive error is therefore not the command itself, but the incorrect assumption that “sudo missing” implies “permission denied”.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reasoning an agent expresses in its trajectory reflects the beliefs that actually guide its subsequent actions, so that an incorrect stated premise can be treated as the cause of the trajectory's divergence.

  • May combine two claims.

    Findingown data

    Epistemic errors consistently account for the largest share of failures in every one of the 21 model-scaffold systems, ranging from 44% for Qwen3 under OpenHands to 80% for Claude under Terminus2.

    “Although the proportion varies substantially from 44% for Qwen3 under OpenHands to 80% for Claude under Terminus2: epistemic errors consistently account for the largest share of failures in every system.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    84% of fabrication begins at or after the lock-in step, and its median onset falls exactly on that step.

    “84% of fabrication begins at or after the lock-in step, and its median onset falls exactly on that step.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Making an error is not what distinguishes successful from failed trajectories; the key difference lies in whether the agent can recover from it.

    “This indicates that making an error is not what distinguishes successful and failed trajectories; instead, the key difference lies in whether the agent can recover from it.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The study's results should be interpreted as a controlled snapshot of three scaffolds and seven frontier models.

    “Our results should therefore be interpreted as a controlled snapshot of three scaffolds and seven frontier models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.

  • May combine two claims.

    Interpretationown data

    In the case study, the information required to avoid the failure was already present in the environment, and the agent simply reasoned from the wrong premise.

    “Importantly, the information required to avoid the failure is already present in the environment-the agent simply reasons from the wrong premise.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reasoning an agent expresses in its trajectory reflects the beliefs that actually guide its subsequent actions, so that an incorrect stated premise can be treated as the cause of the trajectory's divergence.

  • May combine two claims.

    Premisereasoning

    The three selected scaffolds are diverse, lightweight, platform-based, and terminal-native agent designs.

    “Instead, we select these three because they are diverse, lightweight, platform-based, and terminal native agent designs.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [19]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The fix window is hard to exploit automatically: detection mostly confirms failures rather than anticipating them, and improving it depends not only on the monitor but on exposing the goal the agent must be judged against.

    “The fix window is hard to exploit automatically: detection mostly confirms failures rather than anticipating them, and improving it depends not only on the monitor but on exposing the goal the agent must be judged against.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.
    • Rests on an assumption: Terminal-Bench tasks and the three scaffolds/seven models are representative of real-world CLI coding agents.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The study follows the same evaluation protocol as prior work through Harbor, including execution environment, temperature, and maximum number of iterations.

    “To ensure consistency, our experiments follow the same evaluation protocol as the prior work through Harbor [30], including the execution environment, temperature, and maximum number of iterations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Abnormal terminations mainly stem from infrastructure and harness conditions and provide incomplete trajectories that cannot be annotated.

    “Such abnormal terminations mainly stem from infrastructure and harness conditions and provide incomplete trajectories, cannot be annotated, and are out of our study scope of agent behavior.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Every failed trajectory is independently annotated by two annotators.

    “every failed trajectory is independently annotated by two annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Observable failure signals substantially lag behind the actual onset of failure, so analyses based solely on observable outcomes can misidentify when a trajectory truly begins to fail.

    “Observable failure signals substantially lag behind the actual onset of failure.Therefore, analyses based solely on observable outcomes can misidentify when a trajectory truly begins to fail.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

    “suggesting that improving codingagent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.”
    • Also stated in: § A. Results for RQ1 (Error Occurrence) · plain
      “Improving the final success rate requires preventing or detecting decisive errors early in the trajectory, rather than relying on late-stage repair.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.
    • Rests on an assumption: Terminal-Bench tasks and the three scaffolds/seven models are representative of real-world CLI coding agents.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each task is executed under all 3 × 7 scaffold and model combinations.

    “Each task is executed under all 3 × 7 scaffold and model combinations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The study introduces a trajectory annotation framework that decomposes failure into three stages and enables scalable trajectory analysis.

    “We introduce a trajectory annotation framework that decomposes failure into three stages and enables scalable trajectory analysis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    False-premise errors originate from an incorrect assumption about the task or execution environment, whereas competence failures arise because the agent lacks the knowledge required to solve the task.

    “Unlike competence failures, where the agent lacks the knowledge required to solve the task, false-premise errors originate from an incorrect assumption about the task or execution environment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The use of three representative scaffolds and seven frontier models enables broad cross-system comparisons.

    “Our use of three representative scaffolds and seven frontier models enables broad cross-system comparisons.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Inter-annotator agreement was high: Cohen's κ = 0.83 for root causes and weighted κ ≥ 0.94 for the three timepoints.

    “We mitigate annotation subjectivity through a fixed annotation schema and codebook, independent annotation by two of four annotators for every failed trajectory, and high inter-annotator agreement (Cohen's κ = 0.83 for root causes and weighted κ ≥ 0.94 for the three timepoints).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    After exclusion, the aligned set spans 16 task categories and is broadly balanced across task difficulty: 459 easy, 742 medium, and 593 hard trajectories.

    “After exclusion, the aligned set spans 16 task categories and is broadly balanced across task difficulty: 459 easy, 742 medium, and 593 hard trajectories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Across 9,374 trajectories, the widely reported length-failure correlation reverses once task difficulty is controlled, and context gathering and validation separate success from failure.

    “notably, Mehtiyev and Assunc ¸ão [18] show across 9,374 trajectories that the widely reported length-failure correlation reverses once task difficulty is controlled, and that context gathering and validation separate success from failure.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Although Claude models generate draft annotations, every label is verified and finalized by human annotators.

    “Although Claude models generate draft annotations, every label is verified and finalized by human annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Existing trajectory-analysis studies remain partial: some are limited in scale, others characterize failures descriptively without an explanatory structure, and most target code-repository (SWE-bench) settings rather than general terminal interaction.

    “In summary, these studies establish the value of trajectory analysis, yet each remains partial: some are limited in scale [10], [11], others characterize failures descriptively without an explanatory structure [15], and most target coderepository (SWE-bench) settings rather than general terminal interaction [18], [17].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [11]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    Annotation subjectivity was mitigated through a fixed annotation schema and codebook and independent annotation by two of four annotators for every failed trajectory.

    “We mitigate annotation subjectivity through a fixed annotation schema and codebook, independent annotation by two of four annotators for every failed trajectory, and high inter-annotator agreement (Cohen's κ = 0.83 for root causes and weighted κ ≥ 0.94 for the three timepoints).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    Dropping timeout and incomplete trajectories may introduce bias, though such trajectories are outside the study's scope.

    “Dropping timeout and incomplete trajectories may introduce bias, but such trajectories are outside our study scope.”
    Quote in the paperNot a substantial claim
38 of 38

  • The paper relies on this without support.

    The 89 tasks with complete 21-system trajectories are representative of the broader Terminal-Bench task distribution.

    “To enable fair comparisons across scaffold and model combinations, we retain only tasks for which all 21 trajectories are available.This filtering yields 89 tasks, corresponding to a complete evaluation grid of (89 × 21 = 1,869) trajectories.”

  • The paper relies on this without support.

    Terminal-Bench tasks and the three scaffolds/seven models are representative of real-world CLI coding agents.

    “understanding failures in this setting is critical for improving the reliability of real-world coding agents.”

  • The paper relies on this without support.

    The reasoning an agent expresses in its trajectory reflects the beliefs that actually guide its subsequent actions, so that an incorrect stated premise can be treated as the cause of the trajectory's divergence.

    “Once this assumption is adopted, subsequent reasoning remains internally consistent but is built on an incorrect foundation, causing the trajectory to progressively diverge from the correct execution path.”
    “The decisive error is therefore not the command itself, but the incorrect assumption that “sudo missing” implies “permission denied”.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The supposedly complete 21-system task grid has 75 missing runs

    Major

    The filtering rule says every retained task has all 21 trajectories available. Yet the paper says 75 trajectories are unavailable within the resulting 89-task grid, so that grid cannot consist entirely of tasks meeting the stated rule.

  • Successful trajectories are said to recover from errors defined by eventual failure

    Minor

    The paper defines a decisive error as one that determines eventual failure, but later says successful trajectories recover from decisive errors. Under the stated definition, an error followed by successful completion cannot be a decisive error.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "React: Synergizing reasoning and acting in language models," in Proceedings of the International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview. net/forum?id=WE vluYUL-X

Cited by

Still being checked.

Authors
Zhao, X., Li, H., Li, S., Zhao, T., Barr, E. T., Sarro, F., & Ye, H.
Affiliations
  • University College London (GB)
  • Nanjing University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.09510
arXiv
2607.09510
OpenAlex
W7168134016
Cited by
1
References
39
Citations by year
2026: 1
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
fault recovery, epistemic error, agent reliability, OpenHands, software engineering automation, root cause of failure

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review