CoChat
Sign Up
PreprintCited by 26Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

Open access
Deng, Xiang, Da, Jeff, Pan, Edwin, He, Yannis Yiming, Ide, Charles, Garg, Kanak, Lauffer, Niklas, Park, Andrew +14 more·2025·arXiv
DOI 10.48550/arxiv.2509.16941arXiv 2509.16941
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.

AI snapshot

Written from the paper’s full text.

TL;DR

SWE-Bench Pro is a new 1,865-problem benchmark designed to capture long-horizon, enterprise-level software engineering tasks beyond SWE-Bench, with public, held-out, and commercial subsets. Under a unified SWE-Agent scaffold, top coding models resolve below 45% on the public set and below 20% on the commercial set.

Key findings

  • The benchmark contains 1,865 human-verified problems from 41 actively maintained repositories: 731 public problems from 11 repositories, 858 held-out problems from 12 repositories, and 276 commercial problems from 18 proprietary startup repositories. Reference solutions average 107.4 lines of code across 4.1 files; every problem involves at least 10 lines of change, and over 100 tasks demand more than 100 lines.
  • On the public set (N=731), the highest resolve rates are Claude Sonnet 4.5 at 43.6%, Claude Sonnet 4 at 42.7%, OpenAI GPT-5 (High) at 41.8%, and Claude Haiku 4.5 at 39.5%; lower scores include Kimi K2 Instruct at 27.7% and OpenAI GPT-OSS 120B at 16.2%.
  • On the commercial set (N=276), the best models score below 20%: Claude Opus 4.1 at 17.8%, OpenAI GPT-5 (High) at 15.7%, OpenAI GPT-5 (Medium) at 14.9%, Gemini 2.5 Pro Preview at 10.1%, Claude Sonnet 4 at 9.1%, and OpenAI GPT-4o at 3.6%.
  • Removing human augmentations sharply degrades performance: GPT-5 (High) drops from 25.9% with problem statement, requirements, and interface to 8.40% with problem statement only; Claude Opus 4.1 drops from 22.7% to 8.20%. LLM-as-a-judge failure analysis reports that Opus 4.1 primarily fails on wrong solutions (35.9%) and syntax errors (24.2%), while Sonnet 4 shows context overflow (35.6%) and endless file reading (17.0%), and Qwen3 32B has the highest tool error rate (42.0%).

Methods

  • Benchmark construction sourced consecutive commit pairs from 41 repositories; the test patch is the diff of test-related files, and the gold patch is the remaining diff. Human augmentation adds a problem statement, requirements, and optionally an interface; environments are Docker-based with dependency resolution, and tests are human-reviewed fail2pass and pass2pass suites.
  • Evaluation uses the SWE-Agent scaffold with the same default prompt for all models, a maximum of 50 turns, and models as of September 18th, 2025; open-source LLMs are hosted with vllm on a single node with 8 H100 GPUs. Pass@1 resolve rate is reported on the public and commercial sets, while the held-out set remains private.
  • Failure-mode analysis uses GPT-5 as an LLM-as-a-judge on unresolved trajectories, taking the last 20 turns and classifying into hand-curated buckets. An ablation compares the full augmented setting against problem-statement-only.

Limitations

  • The paper states limited language coverage: although SWE-Bench Pro includes Python, JavaScript, TypeScript, and Go, widely used languages such as Java, C++, and Rust are underrepresented.
  • The paper states dependence on the test suite: real software engineering tasks may have multiple correct solutions that do not pass the original tests, and ideally verifiers would accept any valid solution.
  • Evident from the text: only the public set (731 of 1,865 problems) is openly released, while held-out and commercial codebases remain private, limiting full reproducibility. Evaluation is also conducted in a “without any ambiguity” setting that includes problem statement, requirements, and interface, so it measures implementation given detailed specifications rather than ambiguity resolution; the paper notes the setting without requirements is potentially interesting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2509.16941

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 10 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1. Introduction · p. 1

    Many current coding benchmarks are susceptible to contamination because widely used open-source repositories are prime candidates for inclusion in large-scale web-crawled pre-training corpora.

    “First, many benchmarks are susceptible to contamination [7, 13, 15, 20], as exemplified by recent works [5, 7, 15] and social media posts [1, 19]. This risk arises because widely used open-source repositories-particularly those distributed under permissive licenses (e.g., MIT, Apache 2.0, BSD)-are prime candidates for inclusion in the large-scale web-crawled corpora used to pre-train LLMs [3].”
    Quote in the paperSubstantial claim4 of 5 sources checked
    • cites [7] [7] C. Deng, Y. Zhao, X. Tang, M. Gerstein, and A. Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 8706-8719, Mexico City, Mexico, 2024. Association for Computational Linguistics. - source

      The Introduction (page unavailable) says that large-scale crawled web training data may include evaluation benchmarks, supporting the general contamination risk. The paper focuses on NLP benchmarks, not coding benchmarks. The repository-and-license explanation appears in a separate sentence cited to [3], so it is not evidence attributable to this source. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [13] [13] C. White, S. Dooley, ManleyRoberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, M. Goldblum, Abacus.AI, Nyu, and Nvidia. Livebench: A challenging, contamination-free llm benchmark. ArXiv , abs/2406.19314, 2024. URL https://api.semanticscholar.org/CorpusID:270556394 . - source

      The Introduction (p. 1) says many LLM benchmarks are vulnerable to contamination because benchmark questions are published online, and it cites Codeforces problems appearing on GitHub as an example. It does not say that widely used open-source repositories, or repositories with permissive licenses, are especially likely to enter web-crawled pre-training corpora. In the printed citing text, that more specific explanation has a separate citation [3]. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [15] [15] C. Xu, J. Guan, X. Zhao, C. Fu, Q. Xin, Z. Wang, L. Li, J. Fu, H. Wang, and J. Liu. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244 , 2024. - source
    • cites [20]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [3] [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. - source

      Section 2.2 (p. 8) says GPT-3 was trained partly on Common Crawl, and Section 4 (p. 29) identifies benchmark contamination as a risk when training data comes from the internet. The paper does not establish that widely used open-source repositories, including those with permissive licenses, are especially likely to enter those corpora, or address coding benchmarks specifically. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Findingown data
    § Abstract · p. 1

    In evaluation of widely used coding models under a unified scaffold, performance on SWE-BENCH PRO remains below 45% (Pass@1).

    “In our evaluation of widely used coding models, under a unified scaffold, we observe that their performance on SWE-BENCH PRO remains below 45% (Pass@1).”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    SWE-BENCH PRO contains 1,865 problems sourced from 41 actively maintained repositories spanning business applications, B2B services, and developer tools.

    “SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools.”
    Quote in the paperSubstantial claim
  • Findingown data
    § B. Example Task Instance · p. 13

    On the public set of SWE-BENCH PRO (N=731), Claude Opus 4.1 achieved a resolve rate of 22.7%.

    “CLAUDE OPUS 4.1 | 22.7”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 8. Conclusion · p. 9

    Top-tier models like Opus 4.1 and GPT-5 achieve a 23% success rate on SWE-BENCH PRO.

    “top-tier models like Opus 4.1 and GPT-5 achieving a 23% success rate on SWE-BENCH PRO”
    Quote in the paperSubstantial claim
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 34
  • The quote only partly states this claim. 1

  • May combine two claims.

    Interpretationreasoning
    § Abstract · p. 1

    SWE-BENCH PRO is a substantially more challenging benchmark than SWE-Bench, designed to capture realistic, complex, enterprise-level problems beyond SWE-Bench's scope.

    “We introduce SWE-BENCH PRO, a substantially more challenging benchmark that builds upon the best practices of SWE-Bench [19], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-Bench.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [19] [19] C. Zhang et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419 , 2025. source

      Section 3 (p. 3) describes SWE-bench’s real-world issue-resolution task and test-based evaluation, which supports the reference to SWE-bench’s approach. The paper does not assess whether SWE-BENCH PRO builds on that approach; its claims about PRO’s greater difficulty and enterprise-level scope are the citing authors’ own description, not findings established by this source. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 5. Results · p. 6

    Agentless has difficulty in multi-file editing and produces low evaluation scores, so SWE-Agent is used for results.

    “We also explore another popular scaffold, Agentless [14]. However, we find that Agentless has difficulty in multi-file editing, thus, produces low evaluation scores. We focus on SWE-Agent for our results.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] [14] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 , 2024. - source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 6.3 Trajectory Failure Modes · p. 8

    Yang et al. [16] demonstrate 87% alignment of automated judgments with human categorization of failure modes.

    “Our work follows Yang et al. [16], who demonstrate 87% alignment of automated judgments with human categorization of failure modes.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] [16] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Neural Information Processing Systems , 2024. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § Abstract · p. 1

    Problems in the held-out and commercial sets are not publicly accessible, but results on the commercial set are released.

    “Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    SWE-BENCH PRO is partitioned into a public set (11 repositories), a held-out set (12 repositories), and a commercial set (18 proprietary repositories).

    “The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Analysis on Model Performance on SWE-BENCH PRO · p. 7

    Models maintain relatively stable performance for single-file problems but exhibit sharp declines as file count increases; the performance gap between frontier and smaller models widens dramatically beyond 3 files, with Claude Opus 4.1 and OpenAI GPT-5 maintaining above 10% resolve rates even for 10+ files, while open-source alternatives approach near-zero.

    “Model behavior correlates with task complexity. The relationship between file count and resolve rate (Figure 3, top left) reveals distinct performance degradation patterns. Models maintain relatively stable performance for singlefile problems but exhibit sharp declines as file count increases. Notably, the performance gap between frontier and smaller models widens dramatically beyond 3 files, with”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given
    § 3.1 Characteristics of SWE-BENCH PRO · p. 3

    All repositories selected in SWE-BENCH PRO are actively maintained professional projects with substantial user bases, comprehensive documentation, and established development practices.

    “First, all repositories selected in SWE-BENCH PRO are actively maintained professional projects with substantial user bases, comprehensive documentation, and established development practices.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7.1 Limitations · p. 9

    SWE-BENCH PRO's language distribution is not uniform, and widely-used languages like Java, C++, and Rust are underrepresented, which may limit the benchmark's ability to assess agent performance across modern software development.

    “Limited Language Coverage. Although SWE-BENCH PRO includes multiple programming languages (Python, JavaScript, TypeScript, Go), the distribution is not uniform, and some widely-used languages like Java, C++, and Rust are underrepresented. This may limit the benchmark's ability to assess agent performance across the full spectrum of modern software development.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § scale · p. 2

    On average, reference solutions in SWE-BENCH PRO span 107.4 lines of code across 4.1 files; every problem involves at least 10 lines of change, and over 100 tasks demand more than 100 lines of modification.

    “On average, the reference solutions span 107.4 lines of code across 4.1 files. Every problem involves at least 10 lines of change, and over 100 tasks demand more than 100 lines of modification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Trajectory Failure Modes · p. 8

    OPUS 4.1 primarily fails on semantic understanding, with wrong solutions accounting for 35.9% of failures and syntax errors at 24.2%.

    “OPUS 4.1 primarily fails on semantic understanding, with wrong solutions accounting for 35.9% of failures and syntax errors at 24.2%, suggesting strong technical execution but challenges in problem comprehension and algorithmic correctness.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B. Example Task Instance · p. 13

    The public set of SWE-BENCH PRO has N=731.

    “Table 5: Model performance on the public set of SWE-BENCH PRO (N=731).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § scale · p. 2

    SWE-BENCH PRO excludes trivial edits (1-10 lines) and retains only problems requiring substantial, multi-file modifications.

    “To ensure task complexity, we exclude trivial edits (1-10 lines of code) and retain only problems requiring substantial, multi-file modifications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Trajectory Failure Modes · p. 8

    GPT-5 indicates potential differences in effective-tool-use, but fewer wrong solutions.

    “GPT-5 indicates potential differences in effective-tool-use, but fewer wrong solutions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • May combine two claims.

    Findingown data
    § 6.3 Trajectory Failure Modes · p. 8

    SONNET 4 has context overflow as its primary failure mode (35.6%) and substantial endless file reading behaviors (17.0%).

    “SONNET 4 has context overflow as its primary failure mode (35.6%) and substantial endless file reading behaviors (17.0%), suggesting limitations in context management and file navigation strategies.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • The quote only partly states this claim.

    Findingown data
    § 5. Results · p. 6

    On the public set (N=731), CLAUDE SONNET 4.5 achieves 43.6% resolve rate.

    “CLAUDE SONNET 4.5 and CLAUDE SONNET 4 achieve the highest resolve rates at 43.6% and 42.7% respectively, substantially outperforming smaller models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Analysis on Model Performance on SWE-BENCH PRO · p. 7

    Resolve rates vary considerably among repositories; some repositories show consistently low resolve rates (below 10%) across all models, while others allow certain models to exceed 50%.

    “Resolve rate varies across repositories. Figure 3 (right) demonstrates that resolve rates also vary considerably among different repositories in SWE-BENCH PRO. Some repositories show consistently low resolve rates across all models (below 10%), while others allow certain models to achieve resolve rates exceeding 50%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • May combine two claims.

    Findingown data
    § 6.1 Analysis on Model Performance on SWE-BENCH PRO · p. 7

    Resolve rates differ markedly across programming languages; Go and Python generally show higher resolve rates, with some models above 30%, while JavaScript and TypeScript show more variable performance ranging from near 0% to over 30%.

    “Difficulty varies across programming languages. As shown in Figure 3 (left), resolve rates differ markedly across programming languages. Go and Python generally show higher resolve rates across most models, with some models achieving resolve rates above 30% in these languages. JavaScript (JS) and TypeScript (TS) present more variable performance, with resolve rates ranging from near 0% to over 30%”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The public set is representative of the held-out and commercial sets for analysis.

  • May combine two claims.

    Findingown data
    § 3.3 Public, Commercial, and Held-Out SWE-BENCH PRO · p. 4

    The held-out set contains 858 problems mirroring the public set but using a separate set of repositories, kept private to test for overfitting in the future.

    “Held-Out . We hold out a set of 858 problems mirroring the public set but use a separate set of repositories. We keep this set private to test for overfitting in the future.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Public, Commercial, and Held-Out SWE-BENCH PRO · p. 4

    SWE-BENCH PRO consists of 1865 human-verified and augmented problems divided into public, commercial, and held-out subsets.

    “SWE-BENCH PRO consists of a total of 1865 human-verified and augmented problems, divided as three subsets: public, commercial, and held-out.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Public, Commercial, and Held-Out SWE-BENCH PRO · p. 4

    The commercial set contains 276 problems sourced from startup repositories, kept private but with results reported publicly.

    “Commercial . For the commercial set of 276 problems sourced from startup repositories, we keep it private but report results publicly in this paper and will update in the leaderboard.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § scale · p. 2

    Each repository in SWE-BENCH PRO contributes between 50 and 100 instances, with a strict cap of 100 instances, reducing overfitting risk to any single repository.

    “Each repository contributes between 50 and 100 instances, with a strict cap of 100 instances, thereby reducing the risk of overfitting to any single repository.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § Abstract · p. 1

    SWE-BENCH PRO tasks are long-horizon, may require hours to days for a professional software engineer, and often involve patches across multiple files and substantial code modifications.

    “Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Abstract · p. 1

    All SWE-BENCH PRO tasks are human-verified and augmented with sufficient context to ensure resolvability.

    “All tasks are human-verified and augmented with sufficient context to ensure resolvability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human-written requirements and interface specifications are sufficient to make problems self-contained and resolvable.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § B.3 Interface · p. 15

    The interface includes the interfaces for all classes and functions that have been modified or created, including their signatures, and their file path.

    “It includes the interfaces for all classes and functions that have been modified or created, including their signatures, and their file path.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 3.2 Task Specification · p. 3

    SWE-BENCH PRO problems can be more complex than previous SWE-Bench iterations, so requirements are introduced to resolve potential ambiguity.

    “Problems in SWE-BENCH PRO can be more complex than previous iterations of SWE-Bench, and thus, we introduce requirements to resolve any potential ambiguity issues.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given
    § 7.1 Limitations · p. 9

    Real software engineering tasks may have a variety of correct solutions even if they do not pass the original tests outlined in the task.

    “However, real software engineering tasks may have a variety of correct solutions, even if they do not pass the original tests outlined in the task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given
    § 3.1 Characteristics of SWE-BENCH PRO · p. 3

    Each problem in SWE-BENCH PRO goes through a human augmentation and verification process that ensures task descriptions are not missing critical information, tests are well specified, and problems are representative of real-world software engineering tasks.

    “Similar to SWE-Bench Verified, each problem in SWE-BENCH PRO goes through a human augmentation and verification process. This ensures that task descriptions are not missing critical information, tests are well specified to validate the generated solution, and problems are representative of real-world software engineering tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human-written requirements and interface specifications are sufficient to make problems self-contained and resolvable.
    • Rests on an assumption: The benchmark's tasks are representative of real-world software engineering.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 4.2 Creating Task Descriptions · p. 5

    Issue reproduction and problem solving typically requires extended communication among users, contributors, and maintainers, often including screenshots, links, or other media.

    “In practice, issue reproduction and problem solving typically requires extended communication among users, contributors, and codebase maintainers, often including screenshots, links, or other media.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 4.3 Creating Environments · p. 5

    All tests were sent through a human verification pipeline checking relevance and breadth; irrelevant or too-broad tests were dropped, and problems with all such tests were dropped.

    “We additionally send all tests through a human verification pipeline, where each tests is checked if it is relevant to the task description, and if it is not too broad. In either case, we drop tests that fall into either scale category: a) it is irrelevant to the task description, and b) it is too broad. In the case that all tests are too broad or not relevant, we drop the problem.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 4.3 Creating Environments · p. 5

    Automatic verification runs gold tests several times to ensure consistent passage, ensuring environments work properly and no flaky tests remain; problems failing this are dropped.

    “We use automatic verification to ensure that the environment is working as expected. For each environment, we run the gold tests several times and ensure that they pass consistently. This ensures that the environment can be used properly, and also that there are not any flaky tests that may change run by run. We drop any problems that do not pass this criteria.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given
    § scale · p. 15

    Requirements in SWE-BENCH PRO never include specific code implementation and do not leak solutions.

    “Requirements never include specific code implementation and don't leak solutions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § B.3 Interface · p. 15

    The interface field in SWE-BENCH PRO plays an important role in mitigating false negatives for unit test verification.

    “The interface plays an important role in mitigating false negatives for unit test verification.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Task Specification · p. 4

    Every SWE-BENCH PRO task includes human-reviewed test suites with fail2pass tests verifying issue resolution and pass2pass tests ensuring existing functionality remains intact.

    “Every task includes human-reviewed test suites with fail2pass tests that verify issue resolution and pass2pass tests that ensure existing functionalities remain intact.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § B.1 Problem Statement · p. 14

    When creating problem statements, effort is made to keep them as close as possible to the real-world distribution, such as ensuring every problem statement uses the same default issue templates used in the repository for a specific task.

    “When creating problem statements, effort is made to keep the problem statements as close as possible to the real-world distribution, such as ensuring every problem statement uses the same default issue templates that are used in the repository for a specific task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductreasoning
    § B.1.1 Example · p. 14

    Open Library is representative of the kind of repositories SWE-BENCH PRO includes to maximize environment realism.

    “As a real-world full-stack web application, Open Library is representative of the kind of repositories SWE-BENCH PRO includes to maximize environment realism.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § B.1 Problem Statement · p. 14

    Problem statements in SWE-BENCH PRO are curated from existing commits, issues, and PRs in codebases and are rewritten to be well-specified.

    “Problem statements are curated from existing commits, issues, and PRs in codebases, and are rewritten to be well-specified”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 4. Dataset Creation · p. 4

    To ensure faithful and reliable evaluation, the test suite was manually verified and cleaned, and human augmentation of task descriptions was conducted.

    “To ensure a faithful and reliable evaluation, we manually verified and cleaned the test suite, and conduct human augmentation of the task description to include problem statement, requirements and interface that specify all the details necessary to pass the test suite.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 3.2 Task Specification · p. 3

    Explicitly defining class and function names expected by tests avoids false negatives when a model submits a valid solution with different interfaces.

    “Here, we explicitly define the class and function names expected by the tests to avoid this failure mode when relevant.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given
    § 4.2 Creating Task Descriptions · p. 5

    Commit messages are often unstructured, incomplete, or entirely missing.

    “Although metadata are collected during commit scraping, commit messages are often unstructured, incomplete, or entirely missing.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § 4.3 Creating Environments · p. 5

    Environments were constructed manually by software engineering experts, validated by an in-house pipeline for flakiness and golden test passage, and all fail2pass tests were human-verified with irrelevant tests dropped.

    “We create environments through 3 steps: First, we construct environments manually with software engineering experts. Second, we use an in-house pipeline to validate that test are not flaky and that golden tests can pass the test suite successfully. Finally, we have a human-verification of all tests in the fail2pass test list, in which irrelevant tests are dropped.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § B. Example Task Instance · p. 13

    Models are capped at 50 turns and a cost limit of $2 on SWE-BENCH PRO.

    “Models are capped at 50 turns and a cost limit of $2.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 5. Results · p. 6

    Models are evaluated in a setting without ambiguity, including problem statement, requirements, and interface specification in the agent prompt.

    “Models are evaluated in the setting without any ambiguity - that is, we include the problem statement, requirements and interface specification in the agent prompt.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 3.1 Characteristics of SWE-BENCH PRO · p. 3

    Limiting each repository to 50-100+ instances avoids models gaining an advantage by being especially good at a single repository, rewarding generalization.

    “Second, we limit each repository to contribute 50-100+ instances. This avoids the situation where models get an advantage by being especially good at a single repository, rewarding models that can truly generalize.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 8. Conclusion · p. 9

    SWE-BENCH PRO was created by adhering to three core principles: diverse, real-world task selection; challenging, multi-file code changes; and strict contamination prevention.

    “By adhering to three core principles-diverse, real-world task selection; challenging, multi-file code changes; and strict contamination prevention-we have created a benchmark”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.2 Task Specification · p. 4

    Each SWE-BENCH PRO task is evaluated in a containerized, language-specific environment with full dependency resolution, and all environments are released as pre-built docker images to ensure reproducibility.

    “Each task is evaluated in a containerized, language-specific environment with full dependency resolution. Python tasks use isolated virtual environments, JavaScript/TypeScript tasks use Node.js with npm/yarn, and Go tasks use module-aware environments with proper GOPATH configuration. All environments will be released as pre-built docker images to ensure that they are fully reproducible.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § B.3 Interface · p. 15

    The interface is an optional field that is only used when the task solution requires modifying or creating new public interfaces.

    “The interface is an optional field that is only used when the task solution requires modifying or creating new public interfaces.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.2 Task Specification · p. 3

    Requirements in SWE-BENCH PRO are grounded on the unit tests used for validation.

    “These requirements are grounded on the unit tests that are used for validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 4.2 Creating Task Descriptions · p. 5

    The requirements in SWE-BENCH PRO specify expected behavior but do not prescribe how the solution should be implemented.

    “Importantly, the requirements specify the expected behavior but does not prescribe how the solution should be implemented.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 3.1 Characteristics of SWE-BENCH PRO · p. 3

    SWE-BENCH PRO problems are naturally challenging because edits are required to span multiple files and contain substantial code changes.

    “Finally, we require edits to span multiple files and contain a substantial code change, similar to real software engineering tasks. Subsequently, SWE-BENCH PRO problems are naturally challenging.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § B.3 Interface · p. 15

    SWE-BENCH PRO uses unit tests without modification.

    “Since SWE-BENCH PRO uses unit tests without modification”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning
    § scale · p. 10

    SWE-BENCH PRO serves as a robust, contamination-resistant testbed.

    “SWE-BENCH PROserves as a robust, contamination-resistant testbed”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § B.1 Problem Statement · p. 14

    The problem statement structure is similar to a GitHub Issue and includes the same markdown formatting and conventions found in common open-source repositories.

    “The structure of the problem statement is similar to a Github Issue, and includes the same markdown formatting and conventions found in common open-source repositories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § B.2 Requirements · p. 14

    The requirements section includes a list of human-authored requirements that provide additional information the agent needs to create a valid solution verifiable by the unit tests.

    “The requirements section includes a list of human-authored requirements that provide additional information that the agent needs in order to create a valid solution that is verifiable by the unit tests.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § B.3.1 Example · p. 16

    The interfaces are coupled to the associated unit tests implemented in the test patch for the commit.

    “These interfaces are coupled to the associated unit tests implemented in the test patch for this commit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § scale · p. 15

    If a unit test asserts for the presence of a specific error log string, a requirement is written to specify that the solution should produce the exact same error log string.

    “For example, if a unit test asserts for the presence of a specific error log string, a requirement is written to specify that the solution should produce the exact same error log string.”
    Quote in the paperNot a substantial claim
55 of 55

  • The paper relies on this without support.

    Strong copyleft licenses (GPL) create legal barriers to inclusion in commercial training corpora.

    “as the nature of these licenses creates legal barriers to their inclusion in commercial training corpora.”

  • The paper relies on this without support.

    Private commercial repositories are not included in model training data.

    “In addition, we use commercial repositories purchased from startups, which are private.”

  • The paper relies on this without support.

    Human-written requirements and interface specifications are sufficient to make problems self-contained and resolvable.

    “we augment each issue with a list of human-written requirements - simulating the standard engineering practice of resolving issues follow problem specification and provide additional guarantee that the problems are self-contained.”

  • The paper relies on this without support.

    The public set is representative of the held-out and commercial sets for analysis.

    “For all analysis, we use the public set to avoid potential leakage with the commercial set.”

  • The paper relies on this without support.

    The benchmark's tasks are representative of real-world software engineering.

    “problems are representative of real-world software engineering tasks.”

  • The paper relies on this without support.

    The gold patches are correct and optimal solutions.

    “The remaining diff, excluding the test patch, is referred to as the gold patch.”

  • The paper relies on this without support.

    Top-tier models achieve over 70% success rate on benchmarks like SWE-Bench Verified.

    “compared to over 70% on benchmarks like SWE-Bench Verified”

  • The paper relies on this without support.

    The public set of SWE-BENCH PRO (N=731) is representative of the full SWE-BENCH PRO benchmark.

    “Table 5: Model performance on the public set of SWE-BENCH PRO (N=731).”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SWE-Bench Pro has tasks averaging roughly 120 lines of code to solve.

    “SWE-Bench Pro [Deng et al., 2025, Scale AI, 2025], a leading agentic coding benchmark, has tasks averaging roughly 120 lines of code to solve”

    From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Deng et al., 2025

    The scale section (p. 2) reports that SWE-BENCH PRO reference solutions average 107.4 lines of code across 4.1 files. This supports the claim that tasks involve substantial code changes, but the reported average is closer to 107 than to roughly 120 lines. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • SWE-Bench Pro scales the format to 1,865 tasks across 41 repositories, partitioned into public, held-out, and commercial splits; it also augments each mined issue with human-written requirements and an interface specification naming the symbols its tests expect.

    “SWE-Bench Pro [Deng et al., 2025] pushes toward longer-horizon, enterprise-style problems and scales the format to 1,865 tasks across 41 repositories, partitioned into public, held-out, and commercial splits; it also augments each mined issue with human-written requirements and an interface specification naming the symbols its tests expect”

    From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Deng et al., 2025

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • SWE-Bench Pro adds human-written requirements and an interface block that names the classes and functions the inherited tests expect, which reduces false negatives from naming mismatches but does not change how submissions are graded: the grade still comes from the merged pull request's own tests.

    “SWE-Bench Pro [Deng et al., 2025] augments this scheme on the prompt side, adding human-written requirements and an interface block that names the classes and functions the inherited tests expect, which reduces false negatives from naming mismatches but does not change how submissions are graded: the grade still comes from the merged pull request's own tests”

    From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Deng et al., 2025

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • SWE-Bench Pro restricts its public split to copyleft-licensed repositories and reserves commercial and held-out splits sourced from private repositories.

    “SWE-Bench Pro [Deng et al., 2025] restricts its public split to copyleft-licensed repositories and reserves commercial and held-out splits sourced from private repositories.”

    From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Deng et al., 2025

    Section 3.3 (p. 4) says the public set comes from copyleft-licensed repositories and the commercial set from private startup repositories. It also says the held-out set is kept private, but p. 2 specifies that its repositories were selected under strong copyleft licenses; only the commercial set is sourced from proprietary repositories. (LLM, conf=0.94)

    Experimental — accuracy not yet measured

  • Existing public benchmarks are much more concentrated: SWE-Bench Pro Public spans 11 repositories, and SWE-Bench Verified spans 12, with many tasks drawn from prominent, heavily maintained projects.

    “Existing public benchmarks are much more concentrated. SWE-Bench Pro Public spans 11 repositories, and SWE-Bench Verified spans 12, with many tasks drawn from prominent, heavily maintained projects.”

    From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Deng et al., 2025

    The Abstract (p. 1) confirms that SWE-Bench Pro Public draws from 11 repositories. Section 2.1 (p. 2) assigns the figure of 12 repositories to the original SWE-bench, not explicitly to SWE-Bench Verified. Section 3.1 (p. 3) describes Pro’s repositories as actively maintained projects with substantial user bases, but does not establish that description for the benchmarks collectively. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Prose and table give different Opus failure-mode percentages

    Major

    The prose assigns 35.9% of Opus failures to wrong solutions and 24.2% to syntax errors. Table 4 instead reports 50.3% and 31.3%, respectively, among submitted failing trajectories; its counts do not yield the prose percentages when all listed Opus failures are used either.

  • Multi-file requirement conflicts with single-file problems

    Minor

    The paper says edits are required to span multiple files, but its performance analysis discusses single-file problems in the benchmark.

2 of 2

Cites

Flagged first, then by citations

11 of the 42 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., Sampath, K., Krishnan, M., Kundurthy, S., Hendryx, S., Wang, Z., Bharadwaj, V., Holm, J., Aluri, R., Zhang, C. B. C., Jacobson, N., Liu, B., & Kenstler, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2509.16941
arXiv
2509.16941
OpenAlex
W4415252687
Semantic Scholar
4b83aa6340be8e0e59309d37ac4a3c9ae1ece14e
Cited by
4
References
42
Citations by year
2026: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
  • Leibniz international proceedings in informatics (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
LLM agents, failure mode analysis, enterprise software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review