The DOI matches its DataCite record.
SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
Open accessAbstract
The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resolution and feature addition. Existing AI for software engineering leaderboards (e.g., SWE-bench) focus solely on solution accuracy, ignoring the crucial factor of effectiveness in a resource-constrained world. This is a universal problem that also exists beyond software engineering tasks: any AI system should be more than correct - it must also be cost-effective. To address this gap, we introduce SWE-Effi, a set of new metrics to re-evaluate AI systems in terms of holistic effectiveness scores. We define effectiveness as the balance between the accuracy of outcome (e.g., issue resolve rate) and the resources consumed (e.g., token and time). In this paper, we specifically focus on the software engineering scenario by re-ranking popular AI systems for issue resolution on a subset of the SWE-bench benchmark using our new multi-dimensional metrics. We found that AI system's effectiveness depends not just on the scaffold itself, but on how well it integrates with the base model, which is key to achieving strong performance in a resource-efficient manner. We also identified systematic challenges such as the "token snowball" effect and, more significantly, a pattern of "expensive failures". In these cases, agents consume excessive resources while stuck on unsolvable tasks - an issue that not only limits practical deployment but also drives up the cost of failed rollouts during RL training. Lastly, we observed a clear trade-off between effectiveness under the token budget and effectiveness under the time budget, which plays a crucial role in managing project budgets and enabling scalable reinforcement learning, where fast responses are essential.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SWE-Effi explicitly notes scaffold-model entanglement, but changes scaffold without fixing prompt, timeout, and concurrency; its scaffold × model dependency remains a caveat rather than a controlled measurement.
“SWE-Effi [12] explicitly notes scaffold-model entanglement, but changes scaffold without fixing prompt, timeout, and concurrency; its scaffold × model dependency remains a caveat rather than a controlled measurement.”
From Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (2026) · cites this paper as [12]
Section 5.1 (p. 6) explicitly reports that scaffold performance depends on the base model, supporting the first part of the claim. But Section 4 (p. 5) says the study evaluated all scaffold–model combinations on the same issue subset and disabled parallel processing for every scaffold. That directly contradicts the assertions that concurrency was not fixed and that scaffold–model dependency remained merely a caveat rather than a measured comparison; the cited passages do not establish that prompts or timeouts were fixed. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 21
- [1][1] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, 'SWE-bench: Can language models resolve real-world github issues?' in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66 -
- [5][5] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, 'Openhands: An open platform for AI software developers as generalist agents,' in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://openreview.net/forum?id=OJd3ayDDoF -
- [6][6] Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. I. Wang, 'Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution,' arXiv preprint arXiv:2502.18449 , 2025. - · arXiv
- [7][7] Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, 'Autocoderover: Autonomous program improvement,' in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592-1604. -
- [8][8] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, 'Sweagent: Agent-computer interfaces enable automated software engineering,' Advances in Neural Information Processing Systems , vol. 37, pp. 50 528-50 652, 2024. -
- [9][9] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, 'Agentless: Demystifying llm-based software engineering agents,' arXiv preprint arXiv:2407.01489 , 2024. - · arXiv
- [14][14] I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel, 'Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,' 2025. [Online]. Available: https://arxiv.org/abs/2505.20411 - · arXiv
- [15][15] D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li et al. , 'Multi-swe-bench: A multilingual benchmark for issue resolving,' arXiv preprint arXiv:2504.02605 , 2025. - · arXiv
- [16][16] L. Guo, W. Tao, R. Jiang, Y. Wang, J. Chen, X. Liu, Y. Ma, M. Mao, H. Zhang, and Z. Zheng, 'Omnigirl: A multilingual and multimodal benchmark for github issue resolution,' Proceedings of the ACM on Software Engineering , vol. 2, no. ISSTA, pp. 24-46, 2025. -
- [17][17] N. Mündler, M. Müller, J. He, and M. Vechev, 'Swt-bench: Testing and validating real-world bug-fixes with code agents,' Advances in Neural Information Processing Systems , vol. 37, pp. 81 857-81 887, 2024. -
- [18][18] X. He, Q. Liu, M. Du, L. Yan, Z. Fan, Y. Huang, Z. Yuan, and Z. Ma, 'Swe-perf: Can language models optimize code performance on real-world repositories?' 2025. [Online]. Available: https://arxiv.org/abs/2507.12415 - · arXiv
- [19][19] K. Zhao, J. Song, L. Sha, H. Shen, Z. Chen, T. Zhao, X. Liang, and J. Yin, 'Gui testing arena: A unified benchmark for advancing autonomous gui testing agent,' arXiv preprint arXiv:2412.18426 , 2024. - · arXiv
- [20][20] Y. Zhu, A. Kellermann, D. Bowman, P. Li, A. Gupta, A. Danda, R. Fang, C. Jensen, E. Ihli, J. Benn et al. , 'Cve-bench: A benchmark for ai agents' ability to exploit real-world web application vulnerabilities,' arXiv preprint arXiv:2503.17332 , 2025. - · arXiv
- [21][21] A. Antoniades, A. Örwall, K. Zhang, Y. Xie, A. Goyal, and W. Wang, 'Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,' arXiv preprint arXiv:2410.20285 , 2024. - · arXiv
- [22][22] B. Yang, H. Tian, J. Ren, S. Jin, Y . Liu, F. Liu, and B. Le, 'Enhancing repository-level software repair via repository-aware knowledge graphs,' arXiv preprint arXiv:2503.21710 , 2025. - · arXiv
- [23][23] W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng, 'Magis: Llm-based multi-agent framework for github issue resolution,' Advances in Neural Information Processing Systems , vol. 37, pp. 51 963-51 993, 2024. -
- [24][24] H. Su, R. Sun, J. Yoon, P. Yin, T. Yu, and S. Ö. Arık, 'Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments,' arXiv preprint arXiv:2501.10893 , 2025. - · arXiv
- [25][25] P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y. Xiao, Y. Liu, Z. Zhang, J. Chen, C. Gao et al. , 'Trae agent: An llm-based agent for software engineering with test-time scaling,' arXiv preprint arXiv:2507.23370 , 2025. - · arXiv
- [26][26] Y. Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, 'A survey on code generation with llm-based agents,' arXiv preprint arXiv:2508.00083 , 2025. - · arXiv
- [27][27] B. Yang, Z. Cai, F. Liu, B. Le, L. Zhang, T. F. Bissyandé, Y. Liu, and H. Tian, 'A survey of llm-based automated program repair: Taxonomies, design paradigms, and applications,' arXiv preprint arXiv:2506.23749 , 2025. - · arXiv
- [28][28] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, 'React: Synergizing reasoning and acting in language models,' in International Conference on Learning Representations (ICLR) , 2023. -
Cited by
Sorted by citations
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
Self-citationBai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026) · arXiv · doi:10.48550/arxiv.2604.22750 · Cited by 3
“In agentic settings, similar efficiency–performance trade-offs have been observed in long-horizon or ensemble-style systems, where increased computation does not reliably translate to improved task resolution (Fan et al., 2025; Wang et al., 2025a).”
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3
Self-citationClaw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Zheng, M., Han, K., Li, B., Xu, H., Tian, Y., He, W., Zhou, H., Guo, J., Hu, H., Ma, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.12344
“Consistent with resource-aware agent evaluation [6, 15], Claw-SWE-Bench reports task resolved rate , API cost, and average runtime to characterize both the task-solving capabilities and execution efficiency of general-purpose harnesses.”
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Yang, H., Yu, Q., & Desell, T. (2026) · arXiv · doi:10.48550/arxiv.2607.10569
Metadata
- Authors
- Fan, Z., Vasilevski, K., Lin, D., Chen, B., Chen, Y., Zhong, Z., Zhang, J. M., He, P., & Hassan, A. E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.09853
- OpenAlex
- W7081666459
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- coding agents, large language models, effectiveness indicators, resource constraints, token efficiency, problem resolution, SWE-bench
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
