CoChat
Sign Up
PreprintCited by 0Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Open access
Liu, Fengshuo, Liu, Ying, Sun, Ruize, Luo, Lie, Guo, Siyuan·2026·arXiv
DOI 10.48550/arxiv.2609.17394arXiv 2609.17394
Open this paper in CoChat — free accountOpen PDF

Abstract

Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.

AI snapshot

Written from the paper’s full text.

TL;DR

An audit of published SWE-bench per-instance verdicts finds that top coding agents have converged: they solve and fail largely the same instances, so small leaderboard score gaps cannot support strict rank ordering. The paper proposes a five-step resolution audit and argues for reporting comparison-set-specific effective size, model-scaffold provenance, and descriptive tiers instead of ranks.

Key findings

  • On SWE-bench Verified, the top two entries each resolve 396 of 500 instances; the top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Effective size collapses from \(n_{\text{eff}}/n = 0.94\) over all 134 Verified submissions to 0.33 over the top ten and 0.07 over the top two.
  • Frontier solution sets are strongly nested: median nesting is 0.935 against a score-implied baseline of 0.774. The union of instances solved by the top two is 414 versus 396 for the best single system, and 449 for the top ten.
  • Scores depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points for claude-3-5-sonnet (168 to 317 instances), compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction.
  • Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at \(\alpha = 0.05\), while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction.

Methods

  • Audit of 254 public SWE-bench submissions across four splits with per-instance results: Verified (134 submissions, 500 instances), Lite (84, 299), Test (24, 2,294), and Multimodal (12 of 22, 301). No models were run; all analyses use published harness verdicts and metadata.
  • Main analyses compute effective size \(n_{\text{eff}}(S)\), nesting coefficients against score-implied baselines, exact paired McNemar tests for adjacent ranks, Holm correction, bootstrap difference-in-differences for model×scaffold interactions, IRT/KR-20 diagnostics, and paired design arithmetic for required instance budgets.
  • The paper releases the tier partition and a five-step audit protocol: profile shared outcomes, explain nesting, test paired differences, report grouping sensitivity, and estimate the instance budget needed for resolution.

Limitations

  • The paper states that the factorial design is observational: teams choose scaffold and model together, so scaffold effects absorb co-optimisation, engineering effort, and differential model access; 54% of submissions cannot be placed in the design. It also notes one run per submission, contamination/pretraining overlap, intervals that condition on observed submissions and omit execution variability, no equivalence margin, and a pool limited to one benchmark family, one harness, and voluntarily submitted entries.
  • The paper states that non-rejection does not establish equivalence or non-inferiority, tiers are descriptive rather than simultaneous rank intervals, and validation on unrelated leaderboards remains future work.
  • Evident from the text: leaderboard provenance is incomplete and inconsistent—only 61 of 134 Verified submissions carry a single usable model tag, tags vary in naming, and some entries give a URL instead of an identifier—which limits machine-readable reconstruction of the model-scaffold factorial.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2609.17394

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 10 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Over the top two Verified submissions, n_eff/n = 0.07 (36 instances).

    “Over the top two it is 0.07 -36 instances.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.
  • Findingown data

    Over all 134 Verified submissions, n_eff/n = 0.94.

    “Over all 134 Verified submissions n eff /n = 0.94 and the benchmark looks healthy.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.
  • Findingown data

    On SWE-bench Verified, the leading two entries each resolve 396 of 500 instances.

    “On Verified, the leading two entries each resolve 396 of 500 instances.”
    • Also stated in: § 1 Introduction · plain
      “The two leading entries each resolve 396 instances”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.
  • Findingown data

    Nesting exceeds its score-implied baseline in every ability band.

    “Nesting exceeds its score-implied baseline in every band”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.
  • Findingown data

    Over the top ten Verified submissions, n_eff/n = 0.33.

    “Over the top ten it is 0.33”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.
  • May combine two claims. 41
  • Classified as probably not a substantial claim. 12

  • May combine two claims.

    Findingown data

    Six of nine cell-mean interaction tests remain significant after Holm correction, although the observational design does not identify causal scaffold effects.

    “Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects.”
    • Also stated in: § 4.2 The Two Factors Do Not Separate · plain
      “With 100,000 paired bootstrap resamples, six of nine interactions remain significant after Holm correction across all nine tests (α = 0.05)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    A stated leader-based rule yields three descriptive tiers on SWE-bench Verified, or two after Holm correction.

    “A stated leader-based rule yields three descriptive tiers, or two after Holm correction”
    • Also stated in: § Additional information · plain
      “Our rule gives three descriptive tiers, or two after Holm correction”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    The union of instances solved by the top two is 414 against 396 for the best single system, and by the top ten 449.

    “The union of instances solved by the top two is 414 against 396 for the best single system, and by the top ten 449”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    A two-parameter logistic model fitted to the 134 × 468 non-constant response matrix converges cleanly (in-sample accuracy 0.868 against a base rate of 0.550; corr(θ, score) = 0.971; discrimination bounded to [0.1, 4]).

    “A two-parameter logistic model fitted to the 134 × 468 non-constant response matrix converges cleanly (in-sample accuracy 0.868 against a base rate of 0.550; corr(θ, score) = 0.971; discrimination bounded to a conventional [0.1, 4]).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    The excess over baseline is largest in the weakest band (+0.342 against +0.161 at the frontier), while absolute nesting peaks at the frontier.

    “while the excess over baseline is largest in the weakest band (+0.342 against +0.161 at the frontier), absolute nesting peaks exactly where the ranking is read -the frontier”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    For the top ten submissions on SWE-bench Verified, 285 instances are resolved by all ten, 51 by none, and 164 are non-degenerate.

    “The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes.”
    • Also stated in: § 3.1 Most of the Benchmark Cannot Separate the Leaders · plain
      “Over the top ten it is 0.33: 285 instances are solved by all ten, 51 by none, and 164 remain.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    The ten excluded Multimodal submissions record resolved as an integer count rather than a list of instance identifiers, and so cannot support any per-instance analysis.

    “The ten excluded Multimodal submissions record resolved as an integer count rather than a list of instance identifiers, and so cannot support any per-instance analysis”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Submissions span 2023-2025 and range from 2 to 396 resolved instances on Verified.

    “Submissions span 2023-2025 and range from 2 to 396 resolved instances on Verified.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    Rescaling the top thirty onto the 209 instances non-degenerate for the top twenty more than doubles the visible spread, from 8.8 to 18.7 percentage points, and reorders three adjacent pairs, yet separable pairs stay at 0 of 29.

    “Figure 4 rescales the top thirty onto the 209 instances non-degenerate for the top twenty, and the visible spread more than doubles, from 8.8 to 18.7 percentage points, reordering three adjacent pairs -yet separable pairs stay at 0 of 29”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingnone given

    Under independent-sample reasoning at 80% power and a 65% baseline, detectable differences are roughly 23pp at n = 50, 19pp at n = 80, and 8.2pp at n = 500.

    “Under independent-sample reasoning at 80% power and a 65% baseline, detectable differences are roughly 23pp at n = 50, 19pp at n = 80, and 8.2pp at n = 500.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The effect remains large where the design is most current: 99 instances between scaffolds for claude-4-sonnet, over twice the top-thirty spread.

    “the effect remains large where the design is most current: 99 instances between scaffolds for claude-4-sonnet, over twice the top-thirty spread.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    The median gap between adjacent top-thirty pairs is one instance and the median discordance is 61 on Verified and 57 on Lite.

    “The median gap is one instance and the median discordance 61 (Verified) and 57 (Lite)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    The test information function peaks at θ = -0.62, below the median system; the instrument carries 2.3× more information about a median system (I = 278, SE = 0.060) than about one in the top-ten band (I = 118, SE = 0.092).

    “Its test information function peaks at θ = -0.62,below the median system: the instrument carries 2.3× more information about a median system (I = 278, ability SE = 0.060) than about one in the top-ten band (I = 118, SE = 0.092).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    Using the best submission instead gives six uncorrected and five Holm-significant cases.

    “Using the best submission instead gives six uncorrected and five Holm-significant cases”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    Across the 435 pairs in the top thirty, 194 (44.6%) are separable uncorrected and 41 (9.4%) survive Holm-Bonferroni over the whole family.

    “Across the 435 pairs in the top thirty, 194 (44.6%) are separable uncorrected and 41 (9.4%) survive Holm-Bonferroni over the whole family”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    Among the 29 adjacent pairs, the count of separable pairs is 0 either way, and the smallest adjacent p-value is 0.545.

    “but among the 29 adjacent pairs the count is 0 either way, and the smallest adjacent p-value is 0.545”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    A least-squares additive fit on the connected core (20 cells, 10 scaffolds, 7 models) gives R^2 = 0.989 with a fitted scaffold-effect range of 37.6pp against 47.9pp for models, a ratio of 0.78.

    “A least-squares additive fit y = µ + α scaffold + β model on the connected core (20 cells, 10 scaffolds, 7 models) gives R 2 = 0.989 with a fitted scaffold-effect range of 37.6pp against 47.9pp for models, a ratio of 0.78.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    KR-20 over the system×instance matrix is 0.994 across all 134 Verified submissions and 0.940 across the top fifty, but 0.722 across the top twenty and 0.475 across the top ten.

    “KR-20 over the system×instance matrix is 0.994 across all 134 Verified submissions and 0.940 across the top fifty, but 0.722 across the top twenty and 0.475 across the top ten (Table 2).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    epam-ai-run leads sweagent by 95 instances at claude-3-5-sonnet and 51 at claude-4-sonnet (interaction +44, Holm-adjusted p = 0.0035).

    “epam-ai-run leads sweagent by 95 instances at claude-3-5-sonnet and 51 at claude-4-sonnet (interaction +44, Holm-adjusted p = 0.0035).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    On the 2294-instance Test split, whose top 24 range from 0.2% to 52.6%, 14 of 23 adjacent pairs are separable, at a median gap of 42 instances.

    “the 2294-instance Test split, whose top 24 range from 0.2% to 52.6% -14 of 23 adjacent pairs are separable, at a median gap of 42 instances.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingown data

    For claude-4-sonnet (eight scaffolds) the within-model scaffold range is 99 instances; for gpt-4o (six) it is 78.

    “For claude-4-sonnet (eight scaffolds) it is 99 instances; for gpt-4o (six) it is 78.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • May combine two claims.

    Findingreasoning

    For the ten leading adjacent pairs on Verified, excluding two zero-gap pairs, the upper median multiplier is 52× - roughly 26,000 instances of the same character.

    “For the ten leading adjacent pairs on Verified, excluding two zero-gap pairs, the upper median multiplier is 52× -roughly 26,000 instances of the same character.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Automated auditing finds quality problems in over a quarter of SWE-bench tasks and filtering them shifts rankings and raises mean Verified scores by 9.9%.

    “automated auditing finds quality problems in over a quarter of tasks and shows that filtering them shifts rankings and raises mean Verified scores by 9.9% [18]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    Every entry was graded by the same official harness on the same instances, which licenses the paired tests.

    “What makes this pool analysable is that every entry was graded by the same official harness on the same instances, which is what licenses the paired tests below”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The official harness verdicts are accurate and can be taken as given.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Since scaffold and model contribute comparably and 54% of Verified submissions cannot currently be placed in a factorial at all, a structured (model, scaffold, version) field would let anyone reproduce Section 4 without hand normalisation.

    “Since scaffold and model contribute comparably (Section 4) and 54% of Verified submissions cannot currently be placed in a factorial at all, a structured (model, scaffold, version) field -plus attempts, and whether the harness was modified -would let anyone reproduce Section 4 without hand normalisation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    54% of submissions cannot be placed in the factorial design, which skews it towards model generations that several teams have built around.

    “54% of submissions cannot be placed in the design, which also skews it towards model generations that several teams have built around.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Public benchmark instances are exposed to pretraining overlap; tasks mined from private repositories start largely free of it.

    “Public benchmark instances are exposed to pretraining overlap [6]; tasks mined from private repositories start largely free of it, a genuine advantage of an internal suite.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationnone given

    Whether a timed-out run counts as a failure or is excluded moves a score materially, and the choice is invisible in a single number.

    “Whether a timed-out run counts as a failure or is excluded moves a score materially, and the choice is invisible in a single number.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Validation on unrelated leaderboards remains future work; we do not claim that they share the measured magnitudes.

    “validation on unrelated leaderboards remains future work; we do not claim that they share the measured magnitudes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Audits find defects in a substantial share of agentic benchmark items and document lucky passes; a harness that reads a verdict the agent prints rather than the grader's own output detects neither.

    “Audits find defects in a substantial share of agentic benchmark items [18,20] and document lucky passes [13]; a harness that reads a verdict the agent prints rather than the grader's own output detects neither.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [20]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    An instance solved by every system in the frontier set contributes nothing to ordering it, and 285 of 500 are in that state for the top ten.

    “An instance solved by every system in the frontier set contributes nothing to ordering it, and 285 of 500 are in that state for the top ten (Section 3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The fitted scaffold and model ranges are of the same order; scaffolds do not matter more than models.

    “The reading is not that scaffolds matter more than models, but that the fitted ranges are of the same order.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Given only the issue text, models identify the buggy file at 76% accuracy on Verified but 53% on unseen repositories.

    “given only the issue text, models identify the buggy file at 76% accuracy on Verified but 53% on unseen repositories [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    WorkBuddy Bench has code (80 tasks), web (70), security (60) and office (50) subsets.

    “WorkBuddy Bench has code (80 tasks), web (70), security (60) and office (50) subsets [17].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    These thresholds are descriptive references, not a validation of our tier boundaries or evidence of equivalence within tiers.

    “These thresholds are descriptive references, not a validation of our tier boundaries or evidence of equivalence within tiers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    SWE-bench Verified is being read at a resolution it does not have.

    “It is being read at a resolution it does not have.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    An internal suite saturates for the same reason a public one does.

    “re-mine periodically, because an internal suite saturates for the same reason a public one does.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The pool is one benchmark family, one harness, and voluntarily submitted.

    “The pool is one benchmark family, one harness, and voluntarily submitted.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    This two-proportion calculation is not the paired McNemar requirement.

    “This two-proportion calculation is not the paired McNemar requirement”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Our intervals and tests condition on the observed submissions and resample instances; they omit execution variability and do not quantify total uncertainty or prove equivalence.

    “Our intervals and tests condition on the observed submissions and resample instances; they omit execution variability and do not quantify total uncertainty or prove equivalence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    SWE-bench detects many differences across a broader capability range, and its practice of publishing per-instance verdicts for every submission is exactly what made this audit possible.

    “SWE-bench detects many differences across a broader capability range, and its practice of publishing per-instance verdicts for every submission -not universal among leaderboards -is exactly what made this audit possible.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Tasks all candidates pass or fail do not distinguish them.

    “Tasks all candidates pass or fail do not distinguish them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Where defects do exist, the affected instances still occupy every score's denominator, so the measurement stands.

    “where defects do exist [18], the affected instances still occupy every score's denominator, so the measurement stands.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    These are observational associations on a sparse core, not causal effects or fractions of variance explained; team effort and co-optimisation are confounded.

    “These are observational associations on a sparse core, not causal effects or fractions of variance explained; team effort and co-optimisation are confounded.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All inputs are the per-instance evaluation results and metadata that the SWE-bench maintainers publish for every leaderboard submission; no model access, API keys or private data are involved.

    “All inputs are the per-instance evaluation results and metadata that the SWE-bench maintainers publish for every leaderboard submission; no model access, API keys or private data are involved.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Two or three groups arise under these four specifications; this is not a universal upper bound on distinguishable capability levels.

    “Thus two or three groups arise under these four specifications; this is not a universal upper bound on distinguishable capability levels.”
    Quote in the paperSubstantial claim
46 of 46

  • The authors state this as an assumption.

    The official harness verdicts are accurate and can be taken as given.

    “We take harness verdicts as given.”

  • The paper relies on this without support.

    Small differences on coding-agent leaderboards are often read as an ordering of systems.

    “Small differences on coding-agent leaderboards are often read as an ordering of systems.”

  • The paper relies on this without support.

    The human effort estimates released with SWE-bench Verified are valid indicators of task difficulty.

    “The human effort estimates released with SWE-bench Verified argue otherwise”
3 of 3

Cites

Flagged first, then by citations

5 of the 23 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Nunnally, J.C., Bernstein, I.H.: Psychometric Theory. McGraw-Hill, 3 edn. (1994)

Authors
Liu, F., Liu, Y., Sun, R., Luo, L., & Guo, S.
Affiliations
  • Imperial College London (GB)
  • Hong Kong Polytechnic University (HK)
  • Korea University (KR)
  • Jinan University (CN)
ORCID
Liu, Fengshuo
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2609.17394
arXiv
2609.17394
OpenAlex
W7213403503
Semantic Scholar
66d92daaf428e741b3d217274e3001cc139bafc4
Cited by
0
References
23
Type
posted content
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
coding agents, SWE-bench, benchmark evaluation, equivalence testing, McNemar test, Holm correction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review