CoChat
Sign Up
PreprintCited by 0Software Engineering ResearchSoftware System Performance and ReliabilitySoftware Engineering Techniques and Practices

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Open access
Chen, Zhi, Sun, Zhensu, Shi, Yuling, Lo, David, Jiang, Lingxiao·2026·arXiv
DOI 10.48550/arxiv.2607.01211arXiv 2607.01211
Open this paper in CoChat — free accountOpen PDF

Abstract

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper audits three repository-level performance-optimization benchmarks—GSO, SWE-Perf, and SWE-fficiency—and finds that reference-patch validity, scoring rules, and public-submission coverage all complicate leaderboard interpretation. It argues that leaderboard scores alone are insufficient evidence of coding-agent progress.

Key findings

  • Replaying 740 official reference patches across four Google Cloud machine types and three rounds, only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks satisfied their original validity rules in every cross-machine replay. SWE-Perf is especially fragile: its median runtime change is only -0.03%, and 101/138 tasks have median changes within five percentage points of zero.
  • Scoring rules materially shape rankings. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9/28 pairwise comparisons; SWE-fficiency’s worst ten tasks carry 58.5%–82.8% of score weight, and a bounded-penalty diagnostic with floor f = 0.5 changes 6/8 ranks and flips 8/28 pairwise orders.
  • Across 10 public submissions per replay-valid task, at least one submission matches or beats the reference patch on 384/450 (85.3%) GSO and SWE-fficiency tasks, and 449/450 (99.8%) beat the unoptimized base program. The remaining 66 tasks are rarely basic failures: all have a correct public patch, 65/66 beat the base program, and median best-patch speed reaches 85.3% of reference on GSO and 87.9% on SWE-fficiency.
  • Strategy mismatch is not the main remaining gap: 32/66 best public patches use the same high-level optimization category as the reference, while 34/66 use a different category or show no visible production optimization; same-category patches are closer on median (89.8% vs. 81.1%) but the groups overlap substantially.

Methods

  • The authors audit GSO, SWE-Perf, and SWE-fficiency, replaying official reference patches for 740 tasks on four Google Cloud machine profiles (n2, n2d, n4, n4d; 64 vCPUs and 256GB memory) across three rounds, yielding 12 machine-round combinations per task while preserving each benchmark’s official workloads and validity rules. They report faster-than-base and original-rule-valid metrics.
  • For scoring-rule analysis, they use public submissions and per-task outputs released by GSO and SWE-fficiency as of April 30, 2026, compare eight shared submissions, rescore outputs under alternate aggregation rules, compute Spearman rank correlations and pairwise flips, and calculate per-task score-denominator weights; they also test a bounded-penalty diagnostic with floor f = 0.5.
  • For task coverage, they inspect 10 public submissions for each replay-valid task (39 GSO, 411 SWE-fficiency), checking whether any submission passes tests, beats base, or matches/beats the reference patch; for the 66 remaining below-reference tasks, they annotate reference and best public patches using Peng et al.’s optimization taxonomy with GPT-5.5.

Limitations

  • The paper states internal-validity concerns: replays and metric recomputations may be affected by implementation choices, hardware variation, and released artifact quality, and the strict all-replay rule may undercount usable tasks.
  • It states external-validity limits: results cover three benchmarks and specific public leaderboard snapshots, not all coding-agent or performance-engineering settings; the 10-submission analysis is a task-coverage proxy, not a newly engineered multi-agent workflow or a claim about any single submitted configuration, and future models, scaffolds, hardware, and benchmark policies may change the counts.
  • It states data-construction limits: the analysis relies on released tasks, reference patches, public submissions, and scoring records, and SWE-Perf is excluded from the metric and RQ3 task analyses because comparable public agent-output artifacts were unavailable. Evident from the text: the scoring and public-submission coverage analyses therefore apply only to GSO and SWE-fficiency, and the machine profiles and scoring diagnostics are not exhaustive tests of all possible replay or aggregation choices.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.01211

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    At least one public submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks.

    “at least one submission matches or beats the reference patch on 85.3% (384/450) of”
    • Also stated in: § I. INTRODUCTION · plain
      “Across 450 replay-valid GSO and SWE-fficiency tasks, at least one public submission matches or beats the reference patch on 384 tasks”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 10 public submissions per task are representative of the set of top submissions for each task.
  • Findingown data

    With the cap, ranks become closer to median SR and above-reference task counts.

    “With the cap, ranks become closer to median SR and abovereference task counts”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.
  • Findingown data

    449/450 replay-valid tasks have a passing public patch that beats the unmodified base program.

    “Every replay-valid GSO and SWE-fficiency task has at least one passing public patch, and 449/450 tasks have a passing patch that beats the base program.”
    Quote in the paperSubstantial claim
  • Findingown data

    Memory-heavy reference patches are usually approached through a different category (9/12 different).

    “usually approached through another category (9/12 different).”
    Quote in the paperNot a substantial claim
  • Findingown data

    16/32 same-category cases remain below 90% of the reference speedup.

    “16/32 same-category cases still remain below 90% of the reference speedup, while 11/32 different-category cases reach 90-100%.”
    Quote in the paperSubstantial claim
  • May combine two claims. 30
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 13

  • May combine two claims.

    Findingown data

    Worst-1 tasks carry 6.3-33.6%, worst-5 tasks carry 31.4-73.1%, and worst-10 tasks carry 58.5-82.8% of the official score denominator.

    “Figure 3 shows that Worst-1 tasks carry 6.3-33.6%,worst-5 tasks carry 31.4-73.1%,and worst-10 tasks carry 58.5-82.8% of the official score denominator.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

  • May combine two claims.

    Findingown data

    Only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks have reference patches that satisfy their original benchmark validity rules in every cross-machine replay.

    “Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every crossmachine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks”
    • Also stated in: § I. INTRODUCTION · plain
      “Most tasks remain executable, but only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks satisfy their original validity rules in all cross-machine replays.”
    • Also stated in: § C. RQ1.2 Reference Validity Under Replay · plain
      “And, reapplying each benchmark's original validity rule leaves only 39, 11, and 411 valid tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • May combine two claims.

    Findingown data

    8 SWE-fficiency tasks reach only 10-50% of the reference patch's speedup and one is below 10%.

    “The tail still matters, however: 8 SWE-fficiency tasks reach only 10-50% and one is below 10%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Applying SWE-fficiency's harmonic scoring to available GSO per-task submission reports gives a Spearman rank correlation of 0.238 against the official SWE-fficiency ranking and flips 11/28 head-to-head pairs.

    “In the other direction, applying SWE-fficiency's harmonic scoring to available GSO per-task submission reports gives a Spearman rank correlation of 0.238 against the official SWE-fficiency ranking and flips 11/28 head-to-head pairs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

  • May combine two claims.

    Findingown data

    At least one of the 10 public submissions matches the reference patch on 384/450 replay-valid tasks (9/39 GSO, 57/411 SWE-fficiency), leaving 66 tasks below reference speed.

    “at least one of the 10 public submissions matches the reference on 384/450 tasks, leaving 66 tasks not reaching reference speed, 9/39 for GSO and 57/411 for SWE-fficiency.”
    • Also stated in: § Finding RQ3.1 · plain
      “384/450 match or beat the reference patch, all 450 have a passing patch, and only 66 remain below reference.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    32/66 best public patches use the same high-level optimization category as the reference patch, and 34/66 use a different category or show no visible production optimization.

    “Overall, 32/66 best public patches use the same high-level category as the reference patch, and 34/66 use a different category or show no visible production optimization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Under SWE-fficiency's leaderboard scoring rule, a few low-speedup tasks can account for most of the score weight, so a rank difference may reflect broad task performance, a small set of severe failures, or both.

    “Under SWE-fficiency's leaderboard scoring rule, a few low-speedup tasks can account for most of the score weight, so a rank difference may reflect broad task performance, a small set of severe failures, or both.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Algorithm reference patches are the largest category group (31 pairs) and are matched most often (21/31).

    “Algorithm reference patches are the largest group and are matched most often (21/31).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Re-aggregating the same SWE-fficiency task outputs with a GSO-style reference-level pass rate raises Spearman rank correlation with GSO from 0.452 to 0.762 and reduces discordant pairs from 9 to 6.

    “Re-aggregating the same SWE-fficiency task outputs with a GSO-style reference-level pass rate raises Spearman rank correlation with GSO from 0.452 to 0.762 and reduces discordant pairs from 9 to 6.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

  • May combine two claims.

    Findingown data

    Across the 660 attempts behind the 66 remaining tasks, 160 fail functionality or validation, 3 do not complete, and 46 pass correctness without improving runtime.

    “across the 660 attempts behind these 66 tasks, 160 fail functionality or validation, 3 do not complete, and 46 pass correctness without improving runtime.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    SWE-Perf's median within-task standard deviation is 1.41 percentage points, lower than GSO and SWE-fficiency.

    “Its median within-task standard deviation is 1.41 percentage points, lower than GSO and SWEfficiency.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • The quote only partly states this claim.

    Findingown data

    Structure reference patches are split in category alignment (7/16 matched).

    “Structure is split (7/16 matched)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The bounded-penalty diagnostic with floor f = 0.5 changes six of eight submission ranks and flips 8/28 pairwise submission orders relative to the official SWE-fficiency ranking.

    “Table VIII shows that this design choice changes six of eight submission ranks and flips 8/28 pairwise submission orders relative to the official SWE-fficiency ranking.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

  • May combine two claims.

    Findingown data

    A simple bounded-penalty diagnostic changes 6/8 submission ranks and flips 8/28 head-to-head comparisons.

    “A simple boundedpenalty diagnostic changes 6/8 submission ranks and flips 8/28 head-to-head comparisons.”
    • Also stated in: § Results. · plain
      “Table VIII shows that this design choice changes six of eight submission ranks and flips 8/28 pairwise submission orders relative to the official SWE-fficiency ranking.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

  • May combine two claims.

    Findingown data

    The reference patches in only 91/102 GSO tasks, 48/140 SWE-Perf tasks, and 470/498 SWE-fficiency tasks ran faster than the base code across machines.

    “A simple runtime check shows that the reference patches in only 91/102 GSO tasks, 48/140 SWE-Perf tasks, and 470/498 SWE-fficiency tasks ran faster than the base code across machines.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • May combine two claims.

    Findingnone given

    Most reference optimizations are replayable, but a small number fail because external resources, build/evaluation dependencies, or CPU-specific instructions do not port across machines.

    “Most reference optimizations are replayable, but a small number fail because external resources, build/evaluation dependencies, or CPU-specific instructions do not port across machines.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • May combine two claims.

    Findingown data

    For GSO, the best public patch reaches a median 85.3% of the reference patch's speedup and would need 1.17x more speedup to match it.

    “For GSO, the best public patch reaches a median 85.3% of the reference patch's speedup and would need 1.17× more speedup to match it.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After replaying 740 official reference patches across four cloud machines and three rounds, only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks kept their original validity signal in every replay.

    “After replaying 740 official reference patches across four cloud machines and three rounds, only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWEfficiency tasks kept their original validity signal in every replay.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    GSO's scoring rule counts a submitted patch as either a success or a failure.

    “A submitted patch is counted as either a success or a failure [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    SWE-Perf's original validity rule requires each selected efficiency unit test to pass on the original and modified commits, and after 20 timing repetitions and IQR outlier filtering, the recomputed Mann-Whitney minimum-gain value must remain above the dataset threshold: δi > 0.05.

    “SWE-Perf Each selected efficiency unit test must pass on the original and modified commits.After 20 timing repetitions and IQR outlier filtering, the recomputed Mann-Whitney minimum-gain value must remain above the dataset threshold: δi > 0.05.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The replay analysis uses a strict all-replay rule: a task is replay-valid only when the reference patch satisfies the benchmark's original construction rule in all 12 machine-round replays.

    “The replay analysis therefore uses a strict all-replay rule: a task is replay-valid only when the reference patch satisfies the benchmark's original construction rule in all 12 machine-round replays.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GSO's original validity rule requires each selected generated test to still pass correctness/equivalence checks and keep a replay-time speedup of at least 1.2× over the base commit.

    “GSO Each selected generated test must still pass correctness/equivalence checks and keep a replay-time speedup of at least 1.2× over the base commit.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    One likely reason for SWE-Perf's fragility is benchmark construction: SWE-Perf derives efficiency checks from existing repository unit tests and accepts reference gains above a 5% threshold, whereas GSO and SWE-fficiency rely on more explicitly performance-oriented workloads or generated stress tests.

    “One likely reason is benchmark construction: SWE-Perf derives efficiency checks from existing repository unit tests and accepts reference gains above a 5% threshold, whereas GSO and SWE-fficiency rely on more explicitly performance-oriented workloads or generated stress tests.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • Classified as probably not a substantial claim.

    Premisecitation

    SWE-fficiency aggregates SR values with a harmonic mean after flooring each SR at 0.001.

    “The benchmark aggregates SR values with a harmonic mean [25], [26] after flooring each SR at 0.001:”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premiseown data

    As of April 30, 2026, GSO and SWE-fficiency released public ranking data, while SWE-Perf did not have comparable public agent-output data.

    “As of our April 30, 2026 data-collection cutoff, GSO and SWE-fficiency released such public ranking data.SWE-Perf did not have comparable public agent-output data, so it is excluded from this analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    If a new agent configuration is evaluated only by making more tasks reach the reference patch, the remaining room is narrow and the measured improvement may depend heavily on a small set of tasks.

    “If a new agent configuration is evaluated only by making more tasks reach the reference patch, the remaining room is narrow and the measured improvement may depend heavily on a small set of tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The official SWE-fficiency floor f = 0.001 allows one near-zero SR task to contribute 1000 denominator units, or 999 more than an SR = 1 reference-matching task.

    “The official SWE-fficiency floor f = 0.001 allows one near-zero SpeedUp Ratio (SR) task to contribute 1000 denominator units, or 999 more than an SR = 1 reference-matching task.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    SWE-Perf is excluded from the RQ3 task analysis because it has only 11 replay-valid tasks and no comparable released public agent-solution data.

    “SWE-Perf is excluded because it has only 11 replay-valid tasks and no comparable released public agent-solution data.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A reference-level success in GSO is a correct submitted patch whose speedup matches or exceeds the official reference patch.

    “A reference-level success is a correct submitted patch whose speedup matches or exceeds the official reference patch.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Most replay-valid tasks already have a useful public optimization, but that optimization does not yet reach reference-patch speed.

    “Most already have a useful public optimization, but that optimization does not yet reach referencepatch speed.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 10 public submissions per task are representative of the set of top submissions for each task.

  • May combine two claims.

    Interpretationreasoning

    What remains for the unsolved tasks is deeper optimization after the first correct speedup; without seeing the reference patch, runs may stop short of the extra iteration needed to reach reference-patch speed.

    “What remains is deeper optimization after the first correct speedup: without seeing the reference patch, the runs may stop short of the extra iteration needed to reach reference-patch speed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The optimization-category labels assigned to reference and best public patches were manually sanity-checked against patch diffs.

    “We manually sanity-check the labels against patch diffs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The strict all-replay rule may undercount usable tasks but protects later analyses from unstable reference signals.

    “This conservative rule may undercount usable tasks, but it protects later analyses from unstable reference signals.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The results cover three recent repository-level performance-optimization benchmarks and specific public leaderboard snapshots, not all coding-agent or performance-engineering settings.

    “The results cover three recent repositorylevel performance-optimization benchmarks and specific public leaderboard snapshots, not all coding-agent or performanceengineering settings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    SWE-Perf replay failures are mainly a small-signal problem: its reference patches cluster near no runtime change, while GSO and SWE-fficiency reference patches usually have larger speedup margins.

    “SWE-Perf replay failures are mainly a small-signal problem: its reference patches cluster near no runtime change, while GSO and SWE-fficiency reference patches usually have larger speedup margins.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

  • Classified as probably not a substantial claim.

    Premisecitation

    SWE-fficiency compares the submitted patch's speedup with the reference patch's speedup.

    “Instead of a binary gate, SWE-fficiency compares the submitted patch's speedup with the reference patch's speedup [12].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The analysis across 10 public submissions is a task-coverage proxy, not a newly engineered multi-agent workflow or a claim about any single submitted configuration.

    “Our analysis across 10 public submissions is also a task-coverage proxy, not a newly engineered multi-agent workflow or a claim about any single submitted configuration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Code-efficiency benchmarks have expanded from small code units to full repositories.

    “Code-efficiency benchmarks have expanded from small code units to full repositories.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [4]- [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [37]- [41]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Multi-agent workflows can draw on several agent configurations.

    “This matters for multi-agent workflows that can draw on several agent configurations [34], [35].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [34]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

39 of 39

  • The paper relies on this without support.

    Leaderboard scores of repository-level performance-optimization benchmarks are increasingly used as evidence of coding-agent progress.

    “Their leaderboard scores are increasingly used as evidence of coding-agent progress”

  • The paper relies on this without support.

    The four Google Cloud machine profiles used in the replay are representative of the reasonable machine variation that benchmark users encounter in practice.

    “This keeps the cloud provider and resource configuration fixed while varying the kind of hardware difference that benchmark users are likely to encounter in practice (Table II).”

  • The paper relies on this without support.

    The 10 public submissions per task are representative of the set of top submissions for each task.

    “looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of”

  • The paper relies on this without support.

    The official leaderboard scores and ranks released by GSO and SWE-fficiency are accurate.

    “Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons”

  • The paper relies on this without support.

    The best public patch among the 10 submissions (the one with the largest replayed speedup) represents the best available public solution for a task.

    “For each task, we inspect the best public patch: among the 10 public submissions, the patch with the largest replayed speedup relative to the reference patch.”
5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, "SWE-agent: Agent-computer interfaces enable automated software engineering," in Advances in Neural Information Processing Systems, A.
  2. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 50 528-50 652.
  3. D. Huang, Y. Qing, W. Shang, H. Cui, and J. Zhang, "EffiBench: Bench- marking the efficiency of automatically generated code," in Advances in Neural Information Processing Systems, A.
  4. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 11 506-11 544.
  5. M. Du, L. A. Tuan, B. Ji, Q. Liu, and S.-K. Ng, "Mercury: A code efficiency benchmark for code large language models," in Advances in Neural Information Processing Systems, A.
  6. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 16 601-16 622.
  7. M. Shetty, N. Jain, J. Liu, V. Kethanaboyina, K. Sen, and I. Stoica, "GSO: Challenging software optimization tasks for evaluating SWE- agents," in Advances in Neural Information Processing Systems, D. Bel- grave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025.
  8. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, "Autogen: Enabling next-gen llm applications via multi-agent conversation," in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=BAakY1hNKS
  9. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, "React: Synergizing reasoning and acting in language models," in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=WE vluYUL-X
  10. Y. Qing, B. Zhu, M. Du, Z. Guo, T. Y. Zhuo, Q. Zhang, J. M. Zhang, H. Cui, S.-M. Yiu, D. Huang, S.-K. Ng, and A. T. Luu, "EffiBench- X: A multi-language benchmark for measuring efficiency of LLM- generated code," in Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025.

Cited by

Still being checked.

Authors
Chen, Z., Sun, Z., Shi, Y., Lo, D., & Jiang, L.
Affiliations
Singapore Management University (SG)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.01211
arXiv
2607.01211
OpenAlex
W7167043832
Cited by
0
References
46
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
coding agents, benchmark validity, GSO, program optimization, benchmark reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review