The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
Open accessAbstract
Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units.However, real-world software development often involves complex code repositories with complex dependencies and extensive documentation.To enable LLMs to handle these realworld repo-level code generation, we present CODEAGENT, a novel LLM-based agent framework that employs external tools for effective repo-level code generation.CODEAGENT integrates five programming tools, enabling interaction with software artifacts for information retrieval, code implementation, and code testing.We implement four agent strategies to optimize these tools' usage.To the best of our knowledge, CODEAGENT is the first agent framework specifically for repolevel code generation.In order to measure the effectiveness of our method at the repository level, we design a repo-level benchmark CODEAGENTBENCH.The performance on this benchmark shows a significant improvement brought by our method, with improvements in pass rate ranging from 2.0 to 15.8.Further tests on the HumanEval benchmark confirm CODEAGENT's adaptability and efficacy across various code generation tasks.Notably, CODEAGENT outperforms commercial products like GitHub Copilot, showcasing superior accuracy and efficiency.These results demonstrate CODEAGENT's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“CodeAgent [185] shows that real-world repository-level code generation is not simply about completing a single function from a prompt. Instead, the model must locate relevant files, understand dependencies, inspect documentation, implement modifications, and validate outcomes through testing.”
From Code as Agent Harness (2026) · cites this paper as [185]
The cited source says what the claim attributes to it.
In the citing paper
“CodeAgent [114] focuses on the repo-level code generation task, which commonly requires complex dependencies and extensive documentation. It designs a web search strategy for acquiring related external knowledge.”
From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [114]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 563
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
Research works printed but not linked: 20
- Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853.
- Galen Andrew and Jianfeng Gao. 2007. Scalable train- ing of L1-regularized log-linear models. In Proceed- ings of the 24th International Conference on Machine Learning, pages 33-40.
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. CoRR, abs/2304.05128. · arXiv
- Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734.
- Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code gen- eration with large language model. arXiv preprint arXiv:2303.06689. · arXiv
- Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? In The Twelfth International Conference on Learning Representa- tions.
- Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuan- ming Zhang, Yihong Dong, Yuqi Zhu, Bin Gu, and Mengfei Yang. 2024. Deveval: Evaluating code generation in practical software projects. CoRR, abs/2401.06401. · arXiv
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1097.
- Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. arXiv preprint arXiv:2312.05772. Llama. 2023. https://huggingface.co/ meta-llama/Llama-2-70b-chat. · arXiv
- Norman Di Palo, Arunkumar Byravan, Leonard Hasen- clever, Markus Wulfmeier, Nicolas Heess, and Mar- tin A. Riedmiller. 2023. Towards A unified agent with foundation models. CoRR, abs/2307.09668. · arXiv
- Haojie Pan, Zepeng Zhai, Hao Yuan, Yaojia Lv, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. 2023. Kwaiagents: Generalized information-seeking agent system with large language models. CoRR, abs/2312.04889. · arXiv
- Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large lan- guage model connected with massive apis. CoRR, abs/2305.15334. · arXiv
- Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. CoRR, abs/2307.16789. · arXiv
- Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR, abs/2303.17580. · arXiv
- Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023a. A survey on large language model based autonomous agents. CoRR, abs/2308.11432. · arXiv
- Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, and Tao Gui. 2023. The rise and potential of large language model based agents: A survey. CoRR, abs/2309.07864. · arXiv
- John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306.14898. · arXiv
- Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th interna- tional conference on mining software repositories, pages 476-486.
- Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023b. Self-edit: Fault-aware code editor for code genera- tion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 769-787.
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2026) · Data Science and Engineering · doi:10.1007/s41019-025-00335-5 · Cited by 871
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184
(2024) · doi:10.1145/3704435 · Cited by 184
Self-citationA Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 592
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 91
Self-citationAgents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 84
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
Ma, Y., Yang, Q., Cao, R., Li, B., Huang, F., & Li, Y. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728549 · Cited by 114
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Liu, X., Lan, B., Hu, Z., Liu, Y., Zhang, Z., Wang, F., Shieh, M., & Zhou, W. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.7 · Cited by 83
Navigating the Risks: A Survey of Security and Privacy Threats in LLM-Based Agents
Gan, Y., Yang, Y., W, Z., He, P., Zeng, R., Wang, Y., Li, Q., Zhou, C., Li, S., Wang, T., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3807666 · Cited by 112
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 68
ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Lee, H., Oh, S., Kim, J., Shin, J., & Tack, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14565 · Cited by 26
Hu, X. X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 45
RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion
Phan, H. N., Phan, H. N., Nguyen, T. N., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00009 · Cited by 23
SWE-GPT: A Process-Centric Language Model for Automated Software Improvement
Ma, Y., Cao, R., Cao, Y., Zhang, Y., Chen, J., Liu, Y., Liu, Y., Li, B., Huang, F., & Li, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728981 · Cited by 83
Security of LLM-based agents regarding attacks, defenses, and applications: A comprehensive survey
Tang, Y., Liu, Y., Lan, J., Yan, Z., & Erol Gelenbe, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103941 · Cited by 29
DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
Huang, Y., Luo, J., Yu, Y., Zhang, Y., Lei, F., Wei, Y., He, S., Huang, L., Liu, X., Zhao, J., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.748 · Cited by 22
KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding
Xu, Z., Liu, Y., Yin, Y., Zhou, M., & Poovendran, R. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.365 · Cited by 20
Can large language models generate geospatial code?
Hou, S., Shen, Z., Liang, J., Jiao, H., Zhao, A., Qing, Y., Peng, D., Gui, Z., Guan, X., Xiang, L., et al. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2535523 · Cited by 19
GeoCogent: an LLM-based agent for geospatial code generation
Hou, S., Jiao, H., Liang, J., Shen, Z., Zhao, A., & Wu, H. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2549460 · Cited by 18
Qu, Y., Huang, S., Li, Y., Bai, T., Chen, X., Wang, X., Li, L., & Yao, Y. (2025) · Automated Software Engineering · doi:10.1007/s10515-024-00485-2 · Cited by 19
RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation
Li, Q., Xia, W., Dai, X., Du, K., Liu, W., Wang, Y., Tang, R., Yu, Y. J., & Zhang, W. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.410 · Cited by 16
Personal Large Language Model Agents: A Case Study on Tailored Travel Planning
Singh, H., Verma, N., Wang, Y., Bharadwaj, M., Fashandi, H., Ferreira, K., & Chul, L. E. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.37 · Cited by 18
Beyond Browsing: API-Based Web Agents
Song, Y., Xu, F. F., Zhou, S., & Neubig, G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.577 · Cited by 66
Capable language models can outgrow the benefits of collaboration
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01268-y · Cited by 27
KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems
Wang, Z., Yu, B., Zhao, J., Sun, W., Hou, S., Liang, S., Hu, X., Han, Y., & Gan, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11128047 · Cited by 11
BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks
Gandhi, S., Patwardhan, M. S., Vig, L., & Shroff, G. M. (2024) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3703412.3703416 · Cited by 22
AdverIntent-Agent: Adversarial Reasoning for Repair Based on Inferred Program Intent
Ye, H., Yang, A. Z. H., Hu, C., Wang, Y., Zhang, T., & Goues, C. L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728939 · Cited by 11
HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 31
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11
Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
Lu, R., Li, Y., & Huo, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00330 · Cited by 25
Source code vulnerability detection based on deep learning: a review
Su, H., Xu, Z., Zhang, Y., & Tan, Q. (2026) · Cybersecurity · doi:10.1186/s42400-025-00518-7 · Cited by 7
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 9
SEAlign: Alignment Training for Software Engineering Agent
Zhang, K., Zhang, H., Li, G., You, J., Li, J., Zhao, Y., & Jin, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764563 · Cited by 5
Self-citationShift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
Hidayetoğlu, M., Qiao, A., Wyatt, M., Rasley, J., He, Y., & Rajbhandari, S. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790219 · Cited by 10
Situated Instruction Following
Min, S. Y., Puig, X., Chaplot, D. S., Yang, T.-Y., Rai, A., Parashar, P., Salakhutdinov, R., Bisk, Y., & Mottaghi, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_12 · Cited by 8
AutoSilicon: Scaling Up RTL Design Generation Capability of Large Language Models
Li, C., Chen, C., Pan, Y., Xu, W., Liu, Y., Chang, K., Wang, Y., Wang, M., Li, H., Han, Y., et al. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3737286 · Cited by 8
Chen, Y., Guan, W., Azzam, R., & Chen, S. (2025) · AI · doi:10.3390/ai6060127 · Cited by 5
BePilot: An AI Programming Assistant for Compiler Backend Development
Zhong, M., Sun, X., Lv, F., Wang, L., Geng, H., Qiu, L., Cui, H., & Feng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764585 · Cited by 5
Adaptive Model and Strategy Routing for Cost-Efficient LLM Services
Pan, Z., Zhang, K., Zhao, Y., & Han, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792556 · Cited by 5
Self-citationYang, D., Liu, T., Zhang, D., Simoulin, A., Liu, X., Cao, Y., Teng, Z., Qian, X., Yang, G., Luo, J., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.130 · Cited by 9
Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks
Puvvadi, M., Arava, S. K., Santoria, A., Chennupati, S. S. P., & Puvvadi, H. V. (2025) · 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT) · doi:10.1109/csnt64827.2025.10968728 · Cited by 4
Mathew, J. G., & Rossi, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_8 · Cited by 4
TripTailor: A Real-World Benchmark for Personalized Travel Planning
Wang, K., Shen, Y., Lv, C., Zheng, X., & Huang, X. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.503 · Cited by 9
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute
Ma, Y., Li, Y., Dong, Y., Jiang, X., Li, Y., Liu, Y., Cao, R., Chen, J., Huang, F., & Li, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00309 · Cited by 24
MaCTG: Multi-Agent Collaborative Thought Graph for Automatic Programming
Zhao, Z., Sun, J., Hóu, Z., Wei, Z., Cai, C., Qiao, M., & Dong, J. S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773221 · Cited by 9
Large Language Model Agents: A Comprehensive Survey on Architectures, Capabilities, and ApplicationsPreprint
Lei, Y., Xu, J., Liang, C. X., Bi, Z., Li, X., Zhang, D., Song, J., & Yu, Z. (2025) · Preprints.org · doi:10.20944/preprints202512.2119.v1 · Cited by 6
Automating Geospatial Vision Tasks with a Large Language Model Agent
Chen, Y., Wang, W., Kurtz, C., & Lobry, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-662-72243-5_13 · Cited by 3
SGAgent : Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
Zhang, Q., Gao, C., Han, Y., Shang, Y., Fang, C., Chen, Z., & Xiao, L. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3818617 · Cited by 12
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 20
Benchmarking Long-Context Language Models on Long Code Understanding
Li, J., Guo, X., Li, L., Zhang, K., Li, G., Li, J., Tao, Z., Liu, F., Tao, C., Zhu, Y. S., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1324 · Cited by 5
Self-citationLiao, L., Wang, J., Xu, Z., Mu, F., & Yang, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695282 · Cited by 3
Large Language Model-based Data Science Agent: A SurveyPreprint
Wang, P., Yu, Y., Chen, K., Zhan, X., & Wang, H. (2025) · doi:10.36227/techrxiv.175519201.10732484/v1 · Cited by 4
MODERN TRENDS AND CHALLENGES IN DEBUGGING SOFTWARE BASED ON LARGE LANGUAGE MODELS
Zavhorodnii, A., & Ivanov, O. (2025) · International scientific and technical conference Information technologies in metallurgy and machine building · doi:10.34185/1991-7848.itmm.2025.01.039 · Cited by 2
Liu, Z., Liu, S., & Zhao, Y. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343120 · Cited by 2
Liu, C., Lin, Y., Huang, Y., Chang, J., Qi, B., Jiang, B., Huang, Z., & Dong, J. S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00117 · Cited by 2
Zhang, Y., Xia, W., Zhao, B., Yuan, T., & Yu, X. (2026) · Symmetry · doi:10.3390/sym18020248 · Cited by 2
CodeEdu: A Multi-agent Collaborative Platform for Personalized Coding Education
Zhao, J., Gao, P., Cao, J., Wen, Z., Chen, C., Yin, J., Yang, R., & Yuan, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7144-4_24 · Cited by 2
HAFix: history-augmented large language models for bug fixing
Shi, Y., Bangash, A. A., Fallahzadeh, E., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10833-3 · Cited by 8
Towards an Understanding of Context Utilization in Code Intelligence
Wang, Y., Duan, K., Zheng, D., Shi, E., Zhang, F., Wang, Y., Chen, J., Liu, X., Ma, Y., Zhang, H., et al. (2026) · ACM Computing Surveys · doi:10.1145/3797261 · Cited by 2
LUT-LLM: Efficient Language Model Inference with Memory-based Computations on FPGAs
He, Z., Ye, S., Ma, R., Wang, Y., & Cong, J. (2026) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm68464.2026.00027 · Cited by 2
In Line with Context: Repository-Level Code Generation via Context Inlining
Hu, C., Zeng, W., Shi, Y., Shen, B., & Gu, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797094 · Cited by 2
Stenzel, G., Schmid, K., Kölle, M., Altmann, P. J., Lingsch-Rosenfeld, M., Zorn, M., Bücher, T., Gabor, T., Wirsing, M., & Belzner, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75434-0_8 · Cited by 3
LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review ⋆Preprint
Rasheed, Z., Muhammad, W., Kemell, K., Saari, M., & Abrahamsson, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6332149 · Cited by 2
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 197
SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
Tian, M., Wang, Z., Tang, Z., Yang, B., Zhu, K., Dong, H., Li, H., Xie, X., Wang, G., & You, J. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3818488 · Cited by 1
Can an LLM Find Its Way Around a Spreadsheet?
Lee, C.-T., Neeser, A., Xu, S., Katyan, J., Cross, P., Pathakota, S., Norman, M., Simeone, J., Chandrasekaran, J., & Ramakrishnan, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00101 · Cited by 1
Zhang, X., & Yang, B. (2026) · IEEE Access · doi:10.1109/access.2026.3722027 · Cited by 1
Wu, C., Jiang, S., Zuo, Z., & Li, J. (2026) · Applied Sciences · doi:10.3390/app16168291 · Cited by 1
Self-citationA Dual-Model Solution for Business Reasoning with Large Language Models
Trivedi, S., Roy, S. D., Das, R. K. K., Mukherjee, D., Mahapatra, S., & Nandy, P. (2024) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon63413.2024.10895709 · Cited by 1
Exploring Behavior-Driven Development for Code Generation
Liang, Y., Gan, C., Ying, R., & Cui, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_4 · Cited by 1
A Simple Yet Practical Backdoor Prompt Attack Against Black‐Box Code Summarization Engines
Qu, Y., Huang, S., Chen, X., wang, X., & Li, L. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70032 · Cited by 1
AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion
Jiang, T., Wang, Y., Wang, Y., Guo, D., Shi, E., Ma, Y., Chen, T., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00085 · Cited by 1
Huang, Z., Sheng, H., & Qiao, Y. (2025) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc67200.2025.11390477 · Cited by 1
DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generation
Jiao, J., Zhu, H., Yang, P., Wang, J., Liu, J., Liu, Z., Li, D., Fang, Y., Yong, J., Wang, B., et al. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i27.39395 · Cited by 1
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 2
TLR: Codebase-Level C Memory Management Error Repair with Large Language Models
Cheng, X. Y., Guo, Z., Huo, H., & Sui, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797085 · Cited by 1
MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding
Zhang, J. M., Ye, Q., Zhou, H., Liang, H., & Luo, F. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1094 · Cited by 1
Wen, X.-C., Chen, B., Lan, H., Yu, H., Di, P., & Gao, C. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834438 · Cited by 3
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Liu, X., Lan, B., Hu, Z., Liu, Y., Zhang, Z., Wang, F., Shieh, M., & Zhou, W. (2024) · arXiv · doi:10.48550/arxiv.2408.03910 · Cited by 1
Li, Z., Li, Z., Guo, Z., Ren, X., & Huang, C. (2025) · arXiv · doi:10.48550/arxiv.2512.07921
Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167 · Cited by 2
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Self-citationShen, X., Liu, Y., Dai, Y., Wang, Y., Miao, R., Tan, Y., Pan, S., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2505.23352 · Cited by 6
eHAPPA: Efficient and Scalable Resilience Prediction in HPC Applications with Low-Rank AdaptationPreprint
Jiang, H., Zhu, J., Fang, B., BARKER, K. J., Chen, C., Jin, R., & Guan, Q. (2025) · doi:10.36227/techrxiv.175001095.58343307/v1
Repository-Level Code Generation Method Enhanced by Context-Dependent Graph Retrieval
Zhang, H., Li, A., Kuai, Z., Wei, Z., Kuang, L., & Xia, Y. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93257-1_3
Yuma: Geração de Workflows Multiagentes a partir de Linguagem Natural
Neves, A., Mantovani, D., Campelo, C., Souza, G., Souza, D., Furtado, B., Marinho, L. B., & Araújo, E. (2025) · Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia) · doi:10.5753/webmedia_estendido.2025.16421
Liu, S., & Bi, Y. (2025) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4070577
Improving LLM-Assisted Hydropower Operation Through Data Query Enhanced Agent
Luo, W., Zhang, F., Wang, L., Hu, S., Xu, B., & Zhao, J. (2026) · Lecture notes in civil engineering · doi:10.1007/978-981-95-4889-7_49
Hu, H., Yang, X., Wang, J., Zhang, Q., Tang, M., Song, C., Shang, Y., He, C., & Li, W. (2026) · doi:10.36227/techrxiv.177083699.99572896/v1
Collaborative Multi-Agent Framework for Autonomous Code Generation and Debugging
Nazareth, P., Uppoor, P., N, S. H., R, S. S., & Poojary, Y. (2025) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict67193.2025.11399205
LLM Agent Memory: A Survey from a Unified Representation–Management PerspectivePreprint
Tang, Z., He, X., Zhao, T., Wei, F., Liu, X., Dong, P., (32718), Q. W., Li, Q., Wang, H., Chen, R., et al. (2026) · Preprints.org · doi:10.20944/preprints202603.0359.v1
LLMs Can Code, But Can They Do Designƒ
Guruli, A., Gadelia, M., & Tichy, W. F. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418469
Duc, H., Ngo, M. D., Le, T., Le, D.-K., & Le, M. (2026) · Engineering Research Express · doi:10.1088/2631-8695/ae495d
Failure Localization in Multi-Agent Code Generation via Knowledge-Guided and Transferable Reasoning
Geng, M., Gu, S., Liu, Z., Xu, C., Qu, Z., & Wang, H. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i1.36993
LLM Agent Memory: A Survey from a Unified Representation–Management PerspectivePreprint
Tang, Z., He, X., Zhao, T., Wei, F., Liu, X., Dong, P., Wang, Q., Li, Q., Wang, H., Chen, R., et al. (2026) · Preprints.org · doi:10.20944/preprints202603.0359.v2 · Cited by 3
LLM-based multi-agent systems for code generation: A multi-vocal literature review
Rasheeda, Z., Waseema, M., Kemella, K.-K., Saari, M., & Abrahamsson, P. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113142 · Cited by 3
Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
Liu, Z., Huang, Y., Xie, T., (752030), R. Z., Li, W., & Duan, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37314-4_2 · Cited by 1
Metadata
- Authors
- Zhang, K., Li, J., Li, G., Shi, X., & Jin, Z.
- Affiliations
- Peking University (CN)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 13643-13658
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671013
- Cited by
- 138
- References
- 24
- Citations by year
- 2026: 81 · 2025: 47 · 2024: 10
- FWCI
- 44.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61832009, 62192730, 62072007, 62192731, 62192733)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code generation, large language models, LLM agents, tool integration, software repositories, code testing, information retrieval, agent strategies, benchmark, HumanEval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
