CoChat
Sign Up
PreprintCited by 51Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

Open access
Le, Tue, Thai, Minh V. T., Manh, Dung Nguyen, Nhat, Huy Phan, Bui, Nghi D. Q.·2025·arXiv
DOI 10.48550/arxiv.2512.18470arXiv 2512.18470
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.

AI snapshot

Written from the paper’s full text.

TL;DR

SWE-EVO is a benchmark for long-horizon software evolution, built from release transitions in seven mature open-source Python projects, where agents must implement release-level changes across many files rather than fix a single issue. Across 18 models and two scaffolds, the best model resolves only 25% of tasks, and gpt-5.2 drops from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.

Key findings

  • SWE-EVO contains 48 release-sized tasks from 7 Python repositories; tasks average 20.9 edited files, 610.5 edited lines, and 874.0 total tests per instance, with 81.4 FAIL_TO_PASS tests on average.
  • The best model, gpt-5.4, resolves only 25.00% of SWE-EVO under both OpenHands and SWE-agent, while gpt-5.2 falls from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO with SWE-agent.
  • Performance generally scales with model strength, but scaffold sensitivity appears: glm-5 and glm-4p7 perform much better with SWE-agent than OpenHands, reaching 37.50% and 39.58% resolved, respectively.
  • The proposed Fix Rate soft metric reveals partial progress hidden by binary Resolved Rate: under OpenHands, gpt-4.1 and gpt-oss-120b both resolve 2.08%, but their Fix Rates are 4.65% vs. 2.08%.

Methods

  • Benchmark construction used SWE-bench and SWE-Gym as seed pools, selected samples whose base commit matched a repository version tag, defined the problem statement as the release-note delta to the next tagged version, and retained instances with at least one FAIL_TO_PASS test and stable execution; human verification scaled the set to 48 high-confidence instances.
  • Evaluation used OpenHands (CodeActAgent, max 100 iterations) and SWE-agent (max 100 LLM calls) with 18 LLMs from OpenAI, DeepSeek, Zhipu AI, Qwen, and Moonshot AI, using medium reasoning effort for reasoning models and blocking internet access.
  • Primary metrics were Resolved Rate and Patch Apply Rate, plus Fix Rate, a soft metric measuring the fraction of FAIL_TO_PASS tests fixed while assigning 0 if any PASS_TO_PASS test regresses.

Limitations

  • The paper states SWE-EVO covers Python library projects only; specifications are release-note-centered and do not cover all evolution drivers such as security advisories, dependency updates, or design-document-driven refactors.
  • The paper states the benchmark has 48 curated instances with imbalanced repository distribution, especially toward dvc (26 of 48), limiting statistical power for fine-grained comparisons; it also notes Fix Rate weights tests equally and does not measure code quality, maintainability, or patch minimality.
  • The paper states the LLM-as-a-judge failure analysis is qualitative and human agreement validation remains future work; the text also makes evident that one resolved instance changes Resolved Rate by 2.08 percentage points, so close leaderboard gaps should not be over-interpreted.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2512.18470

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 23 cited sources checked · 2 supported · 2 flagged · accuracy not yet measured

  • Premisereasoning

    Prior work on self-evolving agents and tool creation does not specifically target real-world software engineering problems that require long-horizon evolution across multiple commits and versions.

    “While these works demonstrate the potential of self-evolving agents and tool creation, they do not specifically target real-world software engineering problems that require long-horizon evolution across multiple commits and versions.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Robeyns et al., 2025 source

      The Introduction describes SICA as a self-improving coding agent evaluated on SWE Bench Verified, and the Experiments and Results section explicitly calls its benchmark tasks “relatively long-horizon.” That contradicts the claim that this cited work does not target long-horizon software-engineering problems. The paper does not establish that its tasks require evolution across multiple commits and versions. (LLM, conf=0.82)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Zhang et al., 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2025 source

      Section 5 (Related Works; page unavailable) says prior systems DGM and SICA implement self-modifying agents as full software engineering projects, modifying their repositories and validating changes on execution-grounded software engineering tasks. Section 2 also describes their iterative creation of new agent versions. The paper does not establish that these systems address problems spanning multiple commits, but it contradicts the blanket assertion that prior self-evolving agents do not specifically target software engineering problems. (LLM, conf=0.78)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1. Introduction

    In practice, up to 80% of effort is spent maintaining and evolving legacy code, requiring coordinated changes across modules, versions, and specifications.

    “In practice, up to 80% of effort is spent maintaining and evolving legacy code, requiring coordinated changes across modules, versions, and specifications (Kaur and Singh, 2015; Singh et al., 2019).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Kaur and Singh, 2015 source

      Section 2.3, “Change Implementation” (page unavailable), describes changes propagating between components, and section 2.22 discusses maintaining legacy systems. The paper does not establish that up to 80% of effort is spent maintaining and evolving legacy code: its Abstract instead gives an approximate 60% figure for maintenance's share of software cost. The cited passages also do not establish coordinated changes across versions and specifications. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Singh et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisereasoning
    § 2.1. Code Generation and SE Benchmarks

    Prior benchmarks emphasize isolated issue resolution or narrow subskills more than multistep evolution across commits and longer planning horizons.

    “Prior benchmarks therefore emphasize isolated issue resolution or narrow subskills more than multistep evolution across commits and longer planning horizons.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Jimenez et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zan et al., 2025 source

      The paper contrasts benchmarks that isolate code subtasks with SWE-bench’s broader repository-level issue-resolution tasks (passage labeled “Docker container”; page unavailable). Its Introduction describes SWE-bench as generating a patch for an issue, which supports the isolated-issue part of the claim. Our copy does not compare prior benchmarks with multistep evolution across commits or longer planning horizons. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Kuang et al., 2025 source

      Related Work (p. 3) describes early benchmarks as function-level and says most environment-configuration benchmarks evaluate end-to-end executability rather than the process. This supports the contrast with narrow subskills, but the paper does not characterize prior benchmarks as isolated issue resolution or compare them with multistep evolution across commits and longer planning horizons. Its discussion of commits concerns repository selection, not benchmark tasks. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation
    § SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

    Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature.

    “Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature.”
    • Also stated in: § 1. Introduction · plain
      “Early efforts like HumanEval (Chen et al., 2021b) focused on function-level code completion (Chen et al., 2021a), while SWE-Bench (Jimenez et al., 2023) curated real-world GitHub issues requiring verifiable patches for isolated problems (Jimenez et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jimenez et al., 2023 source

      The Introduction (p. 1) says existing coding benchmarks mostly use self-contained problems solvable in a few lines, and Features of SWE-bench (p. 3) says prior settings may limit edits to one function or class. This supports the characterization of isolated tasks, but the paper does not establish that benchmarks for AI coding agents specifically focus on single-issue bug fixes or small feature additions; its discussion of bugs and feature requests describes SWE-bench itself (Section 2, p. 2). (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Findingnone given
    § 1. Introduction

    SWE-EVO uses release notes, commit histories, and versioned snapshots from mature open-source Python projects to construct tasks where agents interpret SRS, plan multi-step modifications, and evolve codebases across releases.

    “SWE-EVO uses release notes, commit histories, and versioned snapshots from mature open-source Python projects to construct tasks where agents interpret Software Requirement Specifications (SRS), plan multi-step modifications, and evolve codebases across releases.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The release-note delta between two versions sufficiently specifies the intended changes for an agent to implement.
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 30

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction

    LLMs have achieved remarkable progress in automating software engineering tasks, including code generation, bug fixing, and test synthesis.

    “Large language models (LLMs) have achieved remarkable progress in automating software engineering (SE) tasks, including code generation (Bui et al., 2023; Chen et al., 2021a; Li et al., 2022; Manh et al., 2023; To et al., 2023; Wang et al., 2023; Wei et al., 2023; Zhuo et al., 2024), bug fixing (Jimenez et al., 2023; Xia et al., 2024), and test synthesis (Chen et al., 2022; Jain et al., 2024; Wang”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Bui et al., 2023 source
    • cites Jimenez et al., 2023 source

      The Introduction (p. 1) describes SWE-bench as an evaluation of language models resolving GitHub issues, typically bug reports, so the paper supports the bug-fixing application cited here. It does not support the characterization of remarkable progress in real-world bug fixing: the Introduction reports that the models failed on all but the simplest issues, with the best resolving 1.96%. The other task categories have separate citations in the citing sentence. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Chen et al., 2022 source

  • May combine two claims.

    Premisecitation
    § 1. Introduction

    SWE-Bench focuses on discrete issue resolution and does not capture the core SE challenge: continuous evolution of existing systems.

    “SWE-Bench focuses on discrete issue resolution and does not capture the core SE challenge: continuous evolution of existing systems (Kaur and Singh, 2015; Singh et al., 2019).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Kaur and Singh, 2015 source

      The Introduction describes maintenance as modifying a software product after delivery to meet new requirements, supporting the importance of evolving existing systems. The paper does not discuss SWE-Bench or establish that discrete issue resolution fails to capture that challenge. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Singh et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction

    Multi-agent systems have rapidly emerged to address long-horizon challenges in software engineering.

    “Building on these capabilities, multi-agent systems have rapidly emerged to address long-horizon challenges in software engineering.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Nguyen et al., 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Phan et al., 2024 source

      The Introduction says LLM-based software agents have emerged to automate complex tasks, including multi-step development processes. The Conclusion describes HyperAgent as a multi-agent system for a wide range of software engineering tasks. The paper does not establish that multi-agent systems as a class have rapidly emerged specifically to address long-horizon challenges. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Wang et al., 2024d

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction

    By coordinating specialized agents for navigation, localization, patching, and verification, multi-agent frameworks improve scalability and performance over single-agent architectures.

    “By coordinating specialized agents for navigation, localization, patching, and verification, these frameworks improve scalability and performance over single-agent architectures.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Nguyen et al., 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Phan et al., 2024 source

      Sections 3.1–3.2 describe HyperAgent’s specialized agents for code navigation, editing, and verification, and say their coordination improves scalability and efficiency. Section 6.1 reports lower resolution rates when child-agent roles are removed. The paper does not establish that multi-agent frameworks generally outperform single-agent architectures in both scalability and performance; that comparison is broader than its findings. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Wang et al., 2024d

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    deepseek-r1 often gets stuck in loops or fails in execution.

    “whereas deepseek-r1 often gets stuck in loops or fails in execution”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM-as-a-judge labels are reliable and valid for diagnosing failure modes.

  • May combine two claims.

    Findingown data
    § (a) GPT-series models.

    In the release-note only setting, performance drops modestly but trends remain consistent, confirming that SWE-EVO poses substantial challenges regardless of specification detail.

    “We also evaluate a more challenging release-note only setting, where agents receive no PR/issue context. As shown in Table 4 (in Appendix E), performance drops modestly but trends remain consistent, confirming that SWE-EVO poses substantial challenges regardless of specification detail.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Release notes frequently refer to specific pull requests or issues.

    “Release notes frequently refer to specific pull requests or issues (e.g., 'see #1234' or 'thanks to PR #5678').”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § (a) GPT-series models.

    Representative 95% Wilson confidence intervals over task instances are [14.9, 38.8] for 25.00%, [10.2, 31.9] for 18.75%, [4.5, 22.2] for 10.42%, and [0.4, 10.9] for 2.08%.

    “Representative 95% Wilson confidence intervals over task instances are [14.9, 38.8] for 25.00%, [10.2, 31.9] for 18.75%, [4.5, 22.2] for 10.42%, and [0.4, 10.9] for 2.08%, so we focus on large performance gaps rather than small ranking differences.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3. Features of SWE-EVO

    SWE-EVO differs from SWE-Bench in presenting substantially richer supervision: longer specifications, broader patches, and heavier test suites.

    “SWE-EVO differs from SWE-Bench along several key dimensions. First, it presents substantially richer supervision: longer specifications, broader patches (more lines, files, and functions edited), and heavier test suites (Figure 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under OpenHands, gpt-4.1 has a Fix Rate of 4.65% and gpt-oss-120b has a Fix Rate of 2.08%.

    “their Fix Rates differ (4.65% vs. 2.08%)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    An exploratory failure analysis suggests that stronger models struggle more with instruction following, while weaker ones exhibit tool-use and syntax errors.

    “An exploratory failure analysis suggests that stronger models struggle more with instruction following, while weaker ones exhibit tool-use and syntax errors.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM-as-a-judge labels are reliable and valid for diagnosing failure modes.

  • May combine two claims.

    Findingown data

    Adding PR/issue context generally improves resolved rates across most models, while preserving the overall ranking and relative trends.

    “As shown in Table 4, adding PR/issue context generally improves resolved rates across most models, while preserving the overall ranking and relative trends.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § (a) GPT-series models.

    Under OpenHands, both gpt-4.1 and gpt-oss-120b resolve only 2.08% of SWE-EVO, but their Fix Rates are 4.65% vs. 2.08%, revealing that gpt-4.1 repairs more failing tests per instance.

    “e.g., under OpenHands, both gpt-4.1 and gpt-oss-120b resolve only 2 . 08% of SWE-EVO, but their Fix Rates are 4 . 65% vs. 2 . 08%, revealing that gpt-4.1 repairs more failing tests per instance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    SWE-EVO's task specifications are release-note-centered, optionally augmented with linked PR/issue text.

    “our task specifications are release-note-centered, optionally augmented with linked PR/issue text”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Stronger models tend to spend more turns on harder instances and terminate earlier on easier ones, while weaker models show less adaptive turn allocation.

    “Panel (b) shows that stronger models tend to spend more turns on harder instances and terminate earlier on easier ones, while weaker models show less adaptive turn allocation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The number of model-scaffold combinations that resolve an instance is a valid measure of empirical difficulty.

  • May combine two claims.

    Findingreasoning
    § 3.3. Features of SWE-EVO

    SWE-EVO instances may aggregate multiple PRs, yielding a wide range of difficulty levels.

    “Second, unlike SWE-Bench where each task maps to a single pull request, SWE-EVO instances may aggregate multiple PRs, yielding a wide range of difficulty levels (see Appendix F).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the GPT series, gpt-5 rarely fails due to syntax or tool issues; over 60% of failures come from Instruction Following.

    “For the GPT series, gpt-5 rarely fails due to syntax or tool issues; instead, over 60% of failures come from Instruction Following”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM-as-a-judge labels are reliable and valid for diagnosing failure modes.

  • May combine two claims.

    Findingown data

    The benchmark has 48 curated instances and an imbalanced repository distribution, especially toward dvc.

    “the benchmark has 48 curated instances and an imbalanced repository distribution, especially toward dvc .”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Figure 6 groups instances by empirical difficulty, using the number of model-scaffold combinations that resolve each instance: r=0, 0<r≤5, 5<r≤10, and r>10, covering roughly 64%, 15%, 15%, and 6% of instances.

    “Figure 6 groups instances by empirical difficulty, using the number of model-scaffold combinations that resolve each instance: 𝑟 = 0, 0 &lt; 𝑟 ≤ 5, 5 &lt; 𝑟 ≤ 10, and 𝑟 &gt; 10, covering roughly 64%, 15%, 15%, and 6% of instances.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The number of model-scaffold combinations that resolve an instance is a valid measure of empirical difficulty.

  • May combine two claims.

    Findingown data

    Experiments with OpenHands and SWE-agent across 18 models reveal a large capability gap: the best model (gpt-5.4) solves only 25% of tasks.

    “Experiments with OpenHands and SWE-agent across 18 models reveal a large capability gap: the best model ( gpt-5.4 ) solves only 25% of tasks”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § (a) GPT-series models.

    One exception is glm-5 / glm-4p7, which perform much better with SWE-agent than OpenHands, suggesting scaffold sensitivity in addition to benchmark difficulty.

    “One exception is glm-5 / glm-4p7 , which perform much better with SWE-agent than OpenHands, suggesting scaffold sensitivity in addition to benchmark difficulty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Fix Rate is execution-verifiable but weights tests equally and does not measure code quality, maintainability, or patch minimality.

    “Fix Rate is execution-verifiable but still weights tests equally and does not measure code quality, maintainability, or patch minimality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Pull Requests per instance

    Stronger models scale turn usage with difficulty, while others show limited adaptivity.

    “(a) instances associated with more pull requests are harder; (b) stronger models scale turn usage with difficulty, while others show limited adaptivity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.1. Benchmark Construction

    Compared with SWE-Bench, SWE-EVO has longer specifications, broader patches, and heavier test suites, making each human-curated release transition substantially larger in engineering scope than a single-issue repair task.

    “Compared with SWE-Bench, SWE-EVO has longer specifications, broader patches, and heavier test suites (Figure 4), making each human-curated release transition substantially larger in engineering scope than a single-issue repair task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1. Introduction

    Recent models reach around 75% on SWE-Bench-Verified and around 40% on the full leaderboard.

    “While recent models reach around 75% on SWE-Bench-Verified (e.g., GPT-5.2 (OpenAI, 2025c)) and around 40% on the full leaderboard (e.g., OpenCSG Starship at 39.67% (Jimenez et al., 2024))”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites OpenAI, 2025c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Jimenez et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

    Fix Rate captures partial progress on complex, long-horizon tasks.

    “We also propose Fix Rate , a metric capturing partial progress on these complex, long-horizon tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 1. Introduction

    Failure analysis suggests that stronger models more often misinterpret nuanced release notes, while weaker models struggle with tool use and syntax errors.

    “Failure analysis suggests that stronger models more often misinterpret nuanced release notes, while weaker models struggle with tool use and syntax errors.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The automatic labeling of failure modes by gpt-5-mini is accurate.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 4.2. Performance on SWE-EVO

    To prevent models from retrieving future code or online PR content during evaluation, we block internet access and provide the linked PR/issue text directly in our default release-note + PR/issue context setting.

    “To prevent models from retrieving future code or online PR content during evaluation, we block internet access and provide the linked PR/issue text directly in our default release-note + PR/issue context setting (Appendix N.1).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Blocking internet access prevents models from retrieving future code or online PR content during evaluation.

  • Classified as probably not a substantial claim.

    Premisenone given

    SWE-EVO captures the software evolution process by tasking agents with evolving codebases between consecutive release versions.

    “SWE-EVO captures this process by tasking agents with evolving codebases between consecutive release versions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    kimi-k2-instruct, qwen3-coder, and gpt-oss-120b have weaker semantic reasoning but stable tool use.

    “indicating weaker semantic reasoning but stable tool use”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM-as-a-judge labels are reliable and valid for diagnosing failure modes.

  • May combine two claims.

    Study design and conductown data
    § 3.1. Benchmark Construction

    After automatic filtering, human effort was used to manually verify task quality and scale the benchmark down to 48 high-confidence instances.

    “After the automatic filtering steps yielded a few hundred candidate release transitions, we used human effort to manually verify task quality and scale the benchmark down to 48 high-confidence instances.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.1. Benchmark Construction

    We discard candidates that trigger installation or runtime errors under the benchmark environment.

    “We additionally discard candidates that trigger installation or runtime errors under the benchmark environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    SWE-EVO requires interpreting release-note-centered specifications, performing multi-file changes, and preserving functionality under comprehensive tests.

    “SWE-EVO requires interpreting release-note-centered specifications, performing multi-file changes, and preserving functionality under comprehensive tests”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 3.2. Task Formulation

    Aggregated across instances, Fix Rate captures partial progress that binary Resolved Rate cannot reflect, while remaining aligned with it.

    “Aggregated across instances, this metric captures partial progress that binary Resolved Rate cannot reflect, while remaining aligned with it (Table 6).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.2. Task Formulation

    Fix Rate is an execution-based progress signal, not a complete measure of code quality, maintainability, or patch minimality.

    “Fix Rate is therefore an execution-based progress signal, not a complete measure of code quality, maintainability, or patch minimality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The strict collection process, requiring stable release tags, reproducible environments, and executable tests, improves validity.

    “Our strict collection process requires stable release tags, reproducible environments, and executable tests, which improves validity”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 3.1. Benchmark Construction

    These differences explain why SWE-EVO stresses release-level software evolution: agents must interpret broader requirements, coordinate changes across more code, and satisfy substantially heavier verification suites.

    “These differences explain why SWE-EVO stresses release-level software evolution: agents must interpret broader requirements, coordinate changes across more code, and satisfy substantially heavier verification suites.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.1. Benchmark Construction

    The quality-first pass checks alignment among the release note, linked PR/issue context, gold patch, and executable tests.

    “This quality-first pass checks alignment among the release note, linked PR/issue context, gold patch, and executable tests.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 3.1. Benchmark Construction

    The framing treats a release as the unit of evolution, matching how users and maintainers typically receive a mixture of fixes, features, and maintenance changes in open-source projects.

    “This framing treats a release as the unit of evolution, matching how users and maintainers typically receive a mixture of fixes, features, and maintenance changes in open-source projects.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A release is the appropriate unit of evolution, matching how users and maintainers typically receive a mixture of fixes, features, and maintenance changes in open-source projects.

  • May combine two claims.

    Interpretationown data
    § (a) GPT-series models.

    SWE-EVO is a more demanding and more realistic benchmark for end-to-end software-evolution capabilities.

    “Taken together, these results indicate that SWE-EVO is a more demanding and more realistic benchmark for end-to-end software-evolution capabilities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 3.1. Benchmark Construction

    Inheriting repositories and execution environments from SWE-bench maximizes reproducibility and comparability and keeps the benchmark plug-and-play for existing SWE agents.

    “To maximize reproducibility and comparability, we inherit repositories and execution environments from SWE-bench and it keeps our benchmark 'plug-and-play' for existing SWE agents.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Jimenez et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

41 of 41

  • The paper relies on this without support.

    Real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality.

    “However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality.”

  • The paper relies on this without support.

    A release is the appropriate unit of evolution, matching how users and maintainers typically receive a mixture of fixes, features, and maintenance changes in open-source projects.

    “This framing treats a release as the unit of evolution, matching how users and maintainers typically receive a mixture of fixes, features, and maintenance changes in open-source projects.”

  • The paper relies on this without support.

    The release-note delta between two versions sufficiently specifies the intended changes for an agent to implement.

    “For each such candidate, we define the problem statement as the release-note delta between that version and its next tagged version; the agent's job is to implement the specified changes across the codebase.”

  • The paper relies on this without support.

    Blocking internet access prevents models from retrieving future code or online PR content during evaluation.

    “To prevent models from retrieving future code or online PR content during evaluation, we block internet access and provide the linked PR/issue text directly in our default release-note + PR/issue context setting (Appendix N.1).”

  • The paper relies on this without support.

    The automatic labeling of failure modes by gpt-5-mini is accurate.

    “Each instance is automatically labeled using gpt-5-mini (OpenAI, 2025b) with categories from Table 8.”

  • The paper relies on this without support.

    The LLM-as-a-judge labels are reliable and valid for diagnosing failure modes.

    “these labels are qualitative diagnostics rather than execution-based scores, and human agreement validation remains future work.”

  • The paper relies on this without support.

    The number of model-scaffold combinations that resolve an instance is a valid measure of empirical difficulty.

    “Figure 6 groups instances by empirical difficulty, using the number of model-scaffold combinations that resolve each instance”

  • The paper relies on this without support.

    Valid software evolution should satisfy the requested change without breaking existing behavior.

    “However, valid software evolution should satisfy the requested change without breaking existing behavior.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Models above 25% contradict the claimed best result

    Major

    The paper calls gpt-5.4 the best model at 25%, but Table 2 reports SWE-agent resolved rates of 37.50% for glm-5 and 39.58% for glm-4p7.

  • Provider model counts total 19, not 18

    Minor

    The stated provider counts add to 19 models, whereas the sentence says 18. The OpenAI list also names nine models rather than the stated ten.

    Reported 18 · recomputed 19

  • A 23.40% resolved rate is incompatible with 48 binary-scored tasks

    Minor

    Resolved Rate counts successfully solved instances out of 48, so it must change in increments of approximately 2.08 percentage points. The reported 23.40% cannot correspond to an integer number of resolved instances.

  • glm-5 average Fix Rate is miscalculated

    Minor

    Table 6 gives glm-5 Fix Rates of 9.67% and 44.27% for the two scaffolds. Their mean is 26.97%, not the reported 27.45%.

    Reported 27.45 · recomputed 26.97

  • o3 SWE-agent resolved rate differs between tables

    Minor

    Table 2 reports 6.25% for o3 with SWE-agent, while Table 6 reports 4.17% for the same model and scaffold.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021b. · arXiv
  2. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=VTF8yNQM66 .
  3. Kimi Team. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 , 2025b. · arXiv
  4. Chamkaur Singh, Neeraj Sharma, and Narender Kumar. Analysis of software maintenance cost affecting factors and estimation models. Int. J. Sci. Technol. Res , 8(9):276-281, 2019.

Cited by

Sorted by citations

Authors
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q.
Affiliations
The University of Melbourne (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2512.18470
arXiv
2512.18470
OpenAlex
W7117153355
Semantic Scholar
03966865097d130afafce247b9fa4cfbac042e60
Cited by
0
References
68
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
coding agents, benchmarking, multi-step reasoning, software release notes, GPT-5, OpenHands, SWE-bench

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review