CoChat
Sign Up
PreprintCited by 1Software Engineering ResearchSoftware Engineering Techniques and PracticesAdvanced Software Engineering Methodologies

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Open access
Orlanski, Gabriel, Roy, Devjeet, Yun, Alexander, Shin, Changho, Gu, Alex, Ge, Albert, Adila, Dyah, Roberts, Nicholas +2 more·2026·arXiv
DOI 10.48550/arxiv.2603.24755arXiv 2603.24755

Preprint — published as 10.5281/zenodo.18405900

Open this paper in CoChat — free account

Abstract

Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.24755

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 55

  1. [1][1] Altaf Allah Abbassi, Leuson Da Silva, Amin Nikanjam, and Foutse Khomh. A taxonomy of inefficiencies in LLM-generated Python code, 2025. URL https://arxiv.org/abs/ 2503.06327 . -
  2. [2][2] Ajmain I. Alam, Palash R. Roy, Farouq Al-Omari, Chanchal K. Roy, Banani Roy, and Kevin A. Schneider. Gptclonebench: A comprehensive benchmark of semantic clones and cross-language clones using gpt-3 model and semanticclonebench. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages 1-13, 2023. doi: 10.1109/ICSME58846. 2023.00013. - · DOI
  3. [3][3] Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992 , 2024. - · arXiv
  4. [9][9] Syed Ammar Asdaque, Imran Haider, Muhammad Umar Malik, Maryam Abdul Ghafoor, and Abdul Ali Bangash. Novice developers produce larger review overhead for project maintainers while vibe coding, 2026. URL https://arxiv.org/abs/2602.23905 . - · arXiv
  5. [10][10] Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, and Alexander Golubev. Swerebench v2: Language-agnostic swe task collection at scale, 2026. URL https://arxiv. org/abs/2602.23866 . -
  6. [11][11] Jeremiah Bohr. Show and tell: Prompt strategies for style control in multi-turn LLM code generation, 2025. URL https://arxiv.org/abs/2511.13972 . - · arXiv
  7. [12][12] G. Ann Campbell. Cognitive complexity - an overview and evaluation. In 2018 IEEE/ACM International Conference on Technical Debt (TechDebt) , pages 57-58, 2018. -
  8. [13][13] Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, and Bing Zhao. Swe-ci: Evaluating agent capabilities in maintaining codebases via continuous integration, 2026. URL https://arxiv. org/abs/2603.03823 . -
  9. [14][14] Yi Chen, Yun Bian, Haiquan Wang, Shihao Li, and Zhe Cui. Scaffold-cegis: Preventing latent security degradation in llm-driven iterative code refinement, 2026. URL https://arxiv. org/abs/2603.08520 . -
  10. [15][15] Zhi Chen and Lingxiao Jiang. Evaluating software development agents: Patch patterns, code quality, and issue complexity in real-world github scenarios. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , pages 657-668, 2025. doi: 10.1109/SANER64311.2025.00068. - · DOI
  11. [16][16] S.R. Chidamber and C.F. Kemerer. A metrics suite for object oriented design. IEEE Transactions on Software Engineering , 20(6):476-493, 1994. doi: 10.1109/32.295895. - · DOI
  12. [17][17] Domenico Cotroneo, Cristina Improta, and Pietro Liguori. Human-written vs. AI-generated code: A large-scale study of defects, vulnerabilities, and complexity, 2025. URL https: //arxiv.org/abs/2508.21634 . -
  13. [18][18] Ward Cunningham. The wycash portfolio management system. volume 4, page 29-30, New York, NY, USA, December 1992. Association for Computing Machinery. doi: 10.1145/157710. 157715. URL https://doi.org/10.1145/157710.157715 . - · DOI
  14. [20][20] Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, and Xingyao Wang. Evoclaw: Evaluating ai agents on continuous software evolution, 2026. URL https://arxiv.org/abs/2603.13428 . - · arXiv
  15. [21][21] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents, 2026. URL https://arxiv.org/abs/2512.12730 . - · arXiv
  16. [22][22] Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Muling Wu, Yunbo Tao, Ming Zhang, Mingxu Chai, Jessica Fan, Zhiheng Xi, Rui Zheng, Yueming Wu, Ming Wen, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. What's wrong with your code generated by large language models? an extensive study. Science China Information Sciences , 2026. doi: 10.1007/ s11432-025-4632-8. URL https://arxiv.org/abs/2407.06153 . - · arXiv
  17. [23][23] Simantika Bhattacharjee Dristi and Matthew B. Dwyer. A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings, 2026. URL https://arxiv. org/abs/2602.15761 . -
  18. [24][24] Guoliang Duan, Mingwei Liu, Yanlin Wang, Chong Wang, Xin Peng, and Zibin Zheng. A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback, 2025. URL https://arxiv.org/abs/2507.00699 . - · arXiv
  19. [27][27] ISO. Iso / iec 25010 : 2011 systems and software engineering - systems and software quality requirements and evaluation ( square ) - system and software quality models, 2013. URL https://api.semanticscholar.org/CorpusID:19022091 . -
  20. [28][28] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=VTF8yNQM66 . -
  21. [29][29] Haolin Jin and Huaming Chen. Are llms reliable code reviewers? systematic overcorrection in requirement conformance judgement, 2026. URL https://arxiv.org/abs/2603. 00539 . -
  22. [30][30] Elmar Juergens, Florian Deissenboeck, Benjamin Hummel, and Stefan Wagner. Do code clones matter? In Proceedings of the 31st International Conference on Software Engineering , ICSE '09, page 485-495, USA, 2009. IEEE Computer Society. ISBN 9781424434534. doi: 10.1109/ ICSE.2009.5070547. URL https://doi.org/10.1109/ICSE.2009.5070547 . - · DOI
  23. [31][31] Kimi Team. Kimi k2.5: Visual agentic intelligence. 2026. URL https://arxiv.org/abs/ 2602.02276 . -
  24. [33][33] Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. Code smells and refactoring: A tertiary systematic review of challenges and observations. Journal of Systems and Software , 167:110610, 2020. ISSN 0164-1212. doi: https://doi.org/10.1016/j. jss.2020.110610. URL https://www.sciencedirect.com/science/article/pii/ S0164121220300881 . - · DOI
  25. [34][34] Duc Minh Le, Suhrid Karthik, Marcelo Schmitt Laser, and Nenad Medvidovic. Architectural decay as predictor of issue- and change-proneness. In 2021 IEEE 18th International Conference on Software Architecture (ICSA) , pages 92-103, 2021. doi: 10.1109/ICSA51549.2021.00017. - · DOI
  26. [35][35] Ruiyin Li, Peng Liang, Mohamed Soliman, and Paris Avgeriou. Understanding software architecture erosion: A systematic mapping study. Journal of Software: Evolution and Process , 34(3):e2423, 2022. doi: https://doi.org/10.1002/smr.2423. URL https://onlinelibrary. wiley.com/doi/abs/10.1002/smr.2423 . - · DOI · Paper page
  27. [36][36] Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025. doi: 10.48550/arXiv. 2503.06680. - · DOI
  28. [38][38] Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, and Ming-Hsuan Yang. Projdevbench: Benchmarking ai coding agents on end-to-end project development, 2026. URL https://arxiv.org/ abs/2602.01655 . - · arXiv
  29. [39][39] T.J. McCabe. A complexity measure. IEEE Transactions on Software Engineering , SE-2(4): 308-320, 1976. doi: 10.1109/TSE.1976.233837. - · DOI · Paper page
  30. [40][40] Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Kwesi Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations , 2026. URL https://openreview.net/forum?id=a7Qa4CcHak . -
  31. [44][44] P. Oman and J. Hagemeister. Metrics for assessing a software system's maintainability. In Proceedings Conference on Software Maintenance 1992 , pages 337-344, 1992. doi: 10.1109/ ICSM.1992.242525. -
  32. [52][52] D.L. Parnas. Software aging. In Proceedings of 16th International Conference on Software Engineering , pages 279-287, 1994. doi: 10.1109/ICSE.1994.296790. - · DOI
  33. [53][53] Norman Peitek, Julia Hess, and Sven Apel. From restructuring to stabilization: A large-scale experiment on iterative code readability refactoring with large language models, 2026. URL https://arxiv.org/abs/2602.21833 . - · arXiv
  34. [54][54] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents, 2025. URL https://arxiv. org/abs/2504.08703 . -
  35. [55][55] César Santos, Ermeson Andrade, and Roberto Natella. Investigating software aging in llmgenerated software systems. In 2025 IEEE 36th International Symposium on Software Reliability Engineering Workshops (ISSREW) , page 314-321. IEEE, October 2025. doi: 10.1109/issrew67781.2025.00090. URL http://dx.doi.org/10.1109/ISSREW67781. 2025.00090 . - · DOI
  36. [56][56] Song Tae-Eun. More rounds, more noise: Why multi-turn review fails to improve cross-context verification, 2026. URL https://arxiv.org/abs/2603.16244 . - · arXiv
  37. [57][57] Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. Sweevo: Benchmarking coding agents in long-horizon software evolution scenarios, 2026. URL https://arxiv.org/abs/2512.18470 . - · arXiv
  38. [58][58] Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe code bench: Evaluating ai models on end-to-end web application development. 2026. URL https: //api.semanticscholar.org/CorpusID:286256067 . -
  39. [59][59] Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, and An Fu. Codeif-bench: Evaluating instruction-following capabilities of large language models in interactive code generation, 2025. URL https://arxiv.org/abs/2503.22688 . - · arXiv
  40. [60][60] Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu Li, Feiyu Xiong, and Wentao Zhang. Codeflowbench: A multi-turn, iterative benchmark for complex code generation, 2026. URL https://arxiv.org/abs/2504.21751 . - · arXiv
  41. [61][61] Zhengren Wang, Rui ling, Chufan Wang, Yongan Yu, Sizhe Wang, Zhiyu li, Feiyu Xiong, and Wentao Zhang. Maintaincoder: Maintainable code generation under dynamic requirements. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2026. URL https://openreview.net/forum?id=TIbQixHEFD . -
  42. [62][62] Kan Watanabe, Tatsuya Shirai, Yutaro Kashiwa, and Hajimu Iida. What to cut? predicting unnecessary methods in agentic code generation, 2026. URL https://arxiv.org/abs/ 2602.17091 . -
  43. [63][63] Jiahong Xiang, Wenxiao He, Xihua Wang, Hongliang Tian, and Yuqun Zhang. Evaluating and improving automated repository-level rust issue resolution with llm-based agents, 2026. URL https://api.semanticscholar.org/CorpusID:286083036 . -
  44. [64][64] Lu Yan, Xuan Chen, and Xiangyu Zhang. When the specification emerges: Benchmarking faithfulness loss in long-horizon coding agents, 2026. URL https://arxiv.org/abs/ 2603.17104 . -
  45. [65][65] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=mXpq6ut8J3 . -
  46. [66][66] Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. Swe-abs: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark, 2026. URL https://arxiv.org/abs/2603.00520 . - · arXiv
  47. [68][68] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, RUI LONG, Ming Ding, and liang xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2026. URL https: //openreview.net/forum?id=MhBZzkz4h9 . -
  48. [69][69] Zhengran Zeng, Yixin Li, Rui Xie, Wei Ye, and Shikun Zhang. Benchmarking and studying the llm-based agent system in end-to-end software development, 2025. URL https://arxiv. org/abs/2511.04064 . -
  49. [70][70] Zexun Zhan, Shuzheng Gao, Ruida Hu, and Cuiyun Gao. Sr-eval: Evaluating llms on code generation under stepwise requirement refinement, 2026. URL https://arxiv.org/abs/ 2509.18808 . -
  50. [71][71] Binquan Zhang, Li Zhang, Lin Shi, Song Wang, Yuwei Qian, Linhui Zhao, Fang Liu, An Fu, and Yida Ye. An empirical study of interaction smells in multi-turn human-llm collaborative code generation, 2026. URL https://arxiv.org/abs/2603.09701 . - · arXiv
  51. [72][72] Zhirui Zhang, Hongbo Zhang, Haoxiang Fei, Zhiyuan Bao, Yubin Chen, Zhengyu Lei, Ziyue Liu, Yixuan Sun, Mingkun Xiao, Zihang Ye, Yu Zhang, Hongcheng Zhu, Yuxiang Wen, and HeungYeung Shum. Swe-agi: Benchmarking specification-driven software construction with moonbit in the era of autonomous agents, 2026. URL https://arxiv.org/abs/2602.09447 . - · arXiv
  52. [73][73] Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. Commit0: Library generation from scratch. 2024. URL https://arxiv. org/abs/2412.01769 . -
  53. [74][74] Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation. 2025. URL https://arxiv.org/abs/2406.06918 . - · arXiv
  54. [75][75] Qixing Zhou, JiaCheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang. Featurebench: Benchmarking agentic coding for complex feature development. In The Fourteenth International Conference on Learning Representations , 2026. URL https://openreview.net/forum? id=41xrZ3uGuI . -
  55. [76][76] Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, 2025. URL https://arxiv. org/abs/2502.06215 .

Cited by

Sorted by citations

Authors
Orlanski, G., Roy, D., Yun, A., Shin, C., Gu, A., Ge, A., Adila, D., Roberts, N., Sala, F., & Albarghouthi, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.24755
arXiv
2603.24755
OpenAlex
W7141610088
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
Keywords
coding agents, iterative software development, long-horizon tasks, redundant codes, architectural design decisions, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review