CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
Open accessAbstract
Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units.However, real-world software development often involves complex code repositories with complex dependencies and extensive documentation.To enable LLMs to handle these realworld repo-level code generation, we present CODEAGENT, a novel LLM-based agent framework that employs external tools for effective repo-level code generation.CODEAGENT integrates five programming tools, enabling interaction with software artifacts for information retrieval, code implementation, and code testing.We implement four agent strategies to optimize these tools' usage.To the best of our knowledge, CODEAGENT is the first agent framework specifically for repolevel code generation.In order to measure the effectiveness of our method at the repository level, we design a repo-level benchmark CODEAGENTBENCH.The performance on this benchmark shows a significant improvement brought by our method, with improvements in pass rate ranging from 2.0 to 15.8.Further tests on the HumanEval benchmark confirm CODEAGENT's adaptability and efficacy across various code generation tasks.Notably, CODEAGENT outperforms commercial products like GitHub Copilot, showcasing superior accuracy and efficiency.These results demonstrate CODEAGENT's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CodeAgent shows that real-world repository-level code generation is not simply about completing a single function from a prompt; instead, the model must locate relevant files, understand dependencies, inspect documentation, implement modifications, and validate outcomes through testing.
“CodeAgent [185] shows that real-world repository-level code generation is not simply about completing a single function from a prompt. Instead, the model must locate relevant files, understand dependencies, inspect documentation, implement modifications, and validate outcomes through testing.”
From Code as Agent Harness (2026) · cites this paper as [185]
CodeAgent focuses on the repo-level code generation task, which commonly requires complex dependencies and extensive documentation; it designs a web search strategy for acquiring related external knowledge.
“CodeAgent [114] focuses on the repo-level code generation task, which commonly requires complex dependencies and extensive documentation. It designs a web search strategy for acquiring related external knowledge.”
From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [114]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Toolformer: language models can teach themselves to use tools
Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2997 · Cited by 1,150 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344 · see citing works
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264 · see citing works
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
Self-citationMapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1192 · Cited by 180 · see citing works
Self-Planning Code Generation with Large Language Models
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., & Jiao, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672456 · Cited by 158 · see citing works
Self-citationYara Parser: A Fast and Accurate Dependency ParserPreprint
Rasooli, M. S., & Tetreault, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.06733 · Cited by 147 · see citing works
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · see citing works
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120 · see citing works
CodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · see citing works
CERT: Continual Pre-training on Sketches for Library-oriented Code Generation
Zan, D., Chen, B., Yang, D., Lin, Z., Kim, M., Guan, B., Wang, Y., Chen, W., & Lou, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/329 · Cited by 85 · see citing works
Multi-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38 · see citing works
ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code GenerationPreprint
Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01861 · Cited by 35 · see citing works
(2023) · doi:10.52202/075280-1035 · Cited by 26 · see citing works
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23 · see citing works
CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained ModelsPreprint
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., & Xie, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00288 · Cited by 19 · see citing works
ToolCoder: Teach Code Generation Models to use API search tools
Zhang, K., Zhang, H., Li, G., Li, J., Li, Z., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.04032 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 20
- Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853.
- Galen Andrew and Jianfeng Gao. 2007. Scalable train- ing of L1-regularized log-linear models. In Proceed- ings of the 24th International Conference on Machine Learning, pages 33-40.
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. CoRR, abs/2304.05128. · arXiv
- Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734.
- Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code gen- eration with large language model. arXiv preprint arXiv:2303.06689. · arXiv
- Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? In The Twelfth International Conference on Learning Representa- tions.
- Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuan- ming Zhang, Yihong Dong, Yuqi Zhu, Bin Gu, and Mengfei Yang. 2024. Deveval: Evaluating code generation in practical software projects. CoRR, abs/2401.06401. · arXiv
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1097.
- Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. arXiv preprint arXiv:2312.05772. Llama. 2023. https://huggingface.co/ meta-llama/Llama-2-70b-chat. · arXiv
- Norman Di Palo, Arunkumar Byravan, Leonard Hasen- clever, Markus Wulfmeier, Nicolas Heess, and Mar- tin A. Riedmiller. 2023. Towards A unified agent with foundation models. CoRR, abs/2307.09668. · arXiv
- Haojie Pan, Zepeng Zhai, Hao Yuan, Yaojia Lv, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. 2023. Kwaiagents: Generalized information-seeking agent system with large language models. CoRR, abs/2312.04889. · arXiv
- Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large lan- guage model connected with massive apis. CoRR, abs/2305.15334. · arXiv
- Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. CoRR, abs/2307.16789. · arXiv
- Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR, abs/2303.17580. · arXiv
- Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023a. A survey on large language model based autonomous agents. CoRR, abs/2308.11432. · arXiv
- Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, and Tao Gui. 2023. The rise and potential of large language model based agents: A survey. CoRR, abs/2309.07864. · arXiv
- John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306.14898. · arXiv
- Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th interna- tional conference on mining software repositories, pages 476-486.
- Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023b. Self-edit: Fault-aware code editor for code genera- tion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 769-787.
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2026) · Data Science and Engineering · doi:10.1007/s41019-025-00335-5 · Cited by 549 · see citing works
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
(2024) · doi:10.1145/3704435 · Cited by 184 · see citing works
Self-citationA Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 65 · see citing works
Self-citationAgents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 50 · see citing works
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
Ma, Y., Yang, Q., Cao, R., Li, B., Huang, F., & Li, Y. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728549 · Cited by 49 · see citing works
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43 · see citing works
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Liu, X., Lan, B., Hu, Z., Liu, Y., Zhang, Z., Wang, F., Shieh, M., & Zhou, W. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.7 · Cited by 38 · see citing works
Navigating the Risks: A Survey of Security and Privacy Threats in LLM-Based Agents
Gan, Y., Yang, Y., W, Z., He, P., Zeng, R., Wang, Y., Li, Q., Zhou, C., Li, S., Wang, T., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3807666 · Cited by 39 · see citing works
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 34 · see citing works
Hu, X. X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 23 · see citing works
RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion
Phan, H. N., Phan, H. N., Nguyen, T. N., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00009 · Cited by 23 · see citing works
SWE-GPT: A Process-Centric Language Model for Automated Software Improvement
Ma, Y., Cao, R., Cao, Y., Zhang, Y., Chen, J., Liu, Y., Liu, Y., Li, B., Huang, F., & Li, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728981 · Cited by 22 · see citing works
Security of LLM-based agents regarding attacks, defenses, and applications: A comprehensive survey
Tang, Y., Liu, Y., Lan, J., Yan, Z., & Erol Gelenbe, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103941 · Cited by 22 · see citing works
DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
Huang, Y., Luo, J., Yu, Y., Zhang, Y., Lei, F., Wei, Y., He, S., Huang, L., Liu, X., Zhao, J., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.748 · Cited by 22 · see citing works
KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding
Xu, Z., Liu, Y., Yin, Y., Zhou, M., & Poovendran, R. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.365 · Cited by 20 · see citing works
Can large language models generate geospatial code?
Hou, S., Shen, Z., Liang, J., Jiao, H., Zhao, A., Qing, Y., Peng, D., Gui, Z., Guan, X., Xiang, L., et al. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2535523 · Cited by 17 · see citing works
GeoCogent: an LLM-based agent for geospatial code generation
Hou, S., Jiao, H., Liang, J., Shen, Z., Zhao, A., & Wu, H. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2549460 · Cited by 17 · see citing works
Qu, Y., Huang, S., Li, Y., Bai, T., Chen, X., Wang, X., Li, L., & Yao, Y. (2025) · Automated Software Engineering · doi:10.1007/s10515-024-00485-2 · Cited by 16 · see citing works
RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation
Li, Q., Xia, W., Dai, X., Du, K., Liu, W., Wang, Y., Tang, R., Yu, Y. J., & Zhang, W. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.410 · Cited by 16 · see citing works
Personal Large Language Model Agents: A Case Study on Tailored Travel Planning
Singh, H., Verma, N., Wang, Y., Bharadwaj, M., Fashandi, H., Ferreira, K., & Chul, L. E. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.37 · Cited by 15 · see citing works
Beyond Browsing: API-Based Web Agents
Song, Y., Xu, F. F., Zhou, S., & Neubig, G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.577 · Cited by 14 · see citing works
Capable language models can outgrow the benefits of collaboration
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01268-y · Cited by 13 · see citing works
KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems
Wang, Z., Yu, B., Zhao, J., Sun, W., Hou, S., Liang, S., Hu, X., Han, Y., & Gan, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11128047 · Cited by 11 · see citing works
BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks
Gandhi, S., Patwardhan, M. S., Vig, L., & Shroff, G. M. (2024) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3703412.3703416 · Cited by 11 · see citing works
AdverIntent-Agent: Adversarial Reasoning for Repair Based on Inferred Program Intent
Ye, H., Yang, A. Z. H., Hu, C., Wang, Y., Zhang, T., & Goues, C. L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728939 · Cited by 11 · see citing works
HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 10 · see citing works
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11 · see citing works
Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
Lu, R., Li, Y., & Huo, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00330 · Cited by 8 · see citing works
Source code vulnerability detection based on deep learning: a review
Su, H., Xu, Z., Zhang, Y., & Tan, Q. (2026) · Cybersecurity · doi:10.1186/s42400-025-00518-7 · Cited by 7 · see citing works
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 6 · see citing works
SEAlign: Alignment Training for Software Engineering Agent
Zhang, K., Zhang, H., Li, G., You, J., Li, J., Zhao, Y., & Jin, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764563 · Cited by 5 · see citing works
Self-citationShift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
Hidayetoğlu, M., Qiao, A., Wyatt, M., Rasley, J., He, Y., & Rajbhandari, S. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790219 · Cited by 5 · see citing works
Situated Instruction Following
Min, S. Y., Puig, X., Chaplot, D. S., Yang, T.-Y., Rai, A., Parashar, P., Salakhutdinov, R., Bisk, Y., & Mottaghi, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_12 · Cited by 5 · see citing works
AutoSilicon: Scaling Up RTL Design Generation Capability of Large Language Models
Li, C., Chen, C., Pan, Y., Xu, W., Liu, Y., Chang, K., Wang, Y., Wang, M., Li, H., Han, Y., et al. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3737286 · Cited by 5 · see citing works
Chen, Y., Guan, W., Azzam, R., & Chen, S. (2025) · AI · doi:10.3390/ai6060127 · Cited by 5 · see citing works
BePilot: An AI Programming Assistant for Compiler Backend Development
Zhong, M., Sun, X., Lv, F., Wang, L., Geng, H., Qiu, L., Cui, H., & Feng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764585 · Cited by 5 · see citing works
Adaptive Model and Strategy Routing for Cost-Efficient LLM Services
Pan, Z., Zhang, K., Zhao, Y., & Han, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792556 · Cited by 5 · see citing works
Self-citationYang, D., Liu, T., Zhang, D., Simoulin, A., Liu, X., Cao, Y., Teng, Z., Qian, X., Yang, G., Luo, J., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.130 · Cited by 5 · see citing works
Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks
Puvvadi, M., Arava, S. K., Santoria, A., Chennupati, S. S. P., & Puvvadi, H. V. (2025) · 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT) · doi:10.1109/csnt64827.2025.10968728 · Cited by 4 · see citing works
Mathew, J. G., & Rossi, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_8 · Cited by 4 · see citing works
TripTailor: A Real-World Benchmark for Personalized Travel Planning
Wang, K., Shen, Y., Lv, C., Zheng, X., & Huang, X. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.503 · Cited by 4 · see citing works
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute
Ma, Y., Li, Y., Dong, Y., Jiang, X., Li, Y., Liu, Y., Cao, R., Chen, J., Huang, F., & Li, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00309 · Cited by 4 · see citing works
MaCTG: Multi-Agent Collaborative Thought Graph for Automatic Programming
Zhao, Z., Sun, J., Hóu, Z., Wei, Z., Cai, C., Qiao, M., & Dong, J. S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773221 · Cited by 4 · see citing works
Large Language Model Agents: A Comprehensive Survey on Architectures, Capabilities, and ApplicationsPreprint
Lei, Y., Xu, J., Liang, C. X., Bi, Z., Li, X., Zhang, D., Song, J., & Yu, Z. (2025) · Preprints.org · doi:10.20944/preprints202512.2119.v1 · Cited by 4 · see citing works
Automating Geospatial Vision Tasks with a Large Language Model Agent
Chen, Y., Wang, W., Kurtz, C., & Lobry, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-662-72243-5_13 · Cited by 3 · see citing works
SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
Zhang, Q., Gao, C., Han, Y., Shang, Y., Fang, C., Chen, Z., & Xiao, L. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3818617 · Cited by 3 · see citing works
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 3 · see citing works
Benchmarking Long-Context Language Models on Long Code Understanding
Li, J., Guo, X., Li, L., Zhang, K., Li, G., Li, J., Tao, Z., Liu, F., Tao, C., Zhu, Y. S., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1324 · Cited by 3 · see citing works
Self-citationLiao, L., Wang, J., Xu, Z., Mu, F., & Yang, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695282 · Cited by 3 · see citing works
Large Language Model-based Data Science Agent: A SurveyPreprint
Wang, P., Yu, Y., Chen, K., Zhan, X., & Wang, H. (2025) · doi:10.36227/techrxiv.175519201.10732484/v1 · Cited by 3 · see citing works
MODERN TRENDS AND CHALLENGES IN DEBUGGING SOFTWARE BASED ON LARGE LANGUAGE MODELS
Zavhorodnii, A., & Ivanov, O. (2025) · International scientific and technical conference Information technologies in metallurgy and machine building · doi:10.34185/1991-7848.itmm.2025.01.039 · Cited by 2 · see citing works
Liu, Z., Liu, S., & Zhao, Y. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343120 · Cited by 2 · see citing works
Liu, C., Lin, Y., Huang, Y., Chang, J., Qi, B., Jiang, B., Huang, Z., & Dong, J. S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00117 · Cited by 2 · see citing works
Zhang, Y., Xia, W., Zhao, B., Yuan, T., & Yu, X. (2026) · Symmetry · doi:10.3390/sym18020248 · Cited by 2 · see citing works
CodeEdu: A Multi-agent Collaborative Platform for Personalized Coding Education
Zhao, J., Gao, P., Cao, J., Wen, Z., Chen, C., Yin, J., Yang, R., & Yuan, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7144-4_24 · Cited by 2 · see citing works
HAFix: history-augmented large language models for bug fixing
Shi, Y., Bangash, A. A., Fallahzadeh, E., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10833-3 · Cited by 2 · see citing works
Towards an Understanding of Context Utilization in Code Intelligence
Wang, Y., Duan, K., Zheng, D., Shi, E., Zhang, F., Wang, Y., Chen, J., Liu, X., Ma, Y., Zhang, H., et al. (2026) · ACM Computing Surveys · doi:10.1145/3797261 · Cited by 2 · see citing works
LUT-LLM: Efficient Language Model Inference with Memory-based Computations on FPGAs
He, Z., Ye, S., Ma, R., Wang, Y., & Cong, J. (2026) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm68464.2026.00027 · Cited by 2 · see citing works
In Line with Context: Repository-Level Code Generation via Context Inlining
Hu, C., Zeng, W., Shi, Y., Shen, B., & Gu, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797094 · Cited by 2 · see citing works
Stenzel, G., Schmid, K., Kölle, M., Altmann, P. J., Lingsch-Rosenfeld, M., Zorn, M., Bücher, T., Gabor, T., Wirsing, M., & Belzner, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75434-0_8 · Cited by 2 · see citing works
LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review ⋆Preprint
Rasheed, Z., Muhammad, W., Kemell, K., Saari, M., & Abrahamsson, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6332149 · Cited by 2 · see citing works
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 1 · see citing works
SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
Tian, M., Wang, Z., Tang, Z., Yang, B., Zhu, K., Dong, H., Li, H., Xie, X., Wang, G., & You, J. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3818488 · Cited by 1 · see citing works
Can an LLM Find Its Way Around a Spreadsheet?
Lee, C.-T., Neeser, A., Xu, S., Katyan, J., Cross, P., Pathakota, S., Norman, M., Simeone, J., Chandrasekaran, J., & Ramakrishnan, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00101 · Cited by 1 · see citing works
Zhang, X., & Yang, B. (2026) · IEEE Access · doi:10.1109/access.2026.3722027 · Cited by 1 · see citing works
Wu, C., Jiang, S., Zuo, Z., & Li, J. (2026) · Applied Sciences · doi:10.3390/app16168291 · Cited by 1 · see citing works
Self-citationA Dual-Model Solution for Business Reasoning with Large Language Models
Trivedi, S., Roy, S. D., Das, R. K. K., Mukherjee, D., Mahapatra, S., & Nandy, P. (2024) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon63413.2024.10895709 · Cited by 1 · see citing works
Exploring Behavior-Driven Development for Code Generation
Liang, Y., Gan, C., Ying, R., & Cui, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_4 · Cited by 1 · see citing works
A Simple Yet Practical Backdoor Prompt Attack Against Black‐Box Code Summarization Engines
Qu, Y., Huang, S., Chen, X., wang, X., & Li, L. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70032 · Cited by 1 · see citing works
AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion
Jiang, T., Wang, Y., Wang, Y., Guo, D., Shi, E., Ma, Y., Chen, T., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00085 · Cited by 1 · see citing works
Huang, Z., Sheng, H., & Qiao, Y. (2025) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc67200.2025.11390477 · Cited by 1 · see citing works
DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generation
Jiao, J., Zhu, H., Yang, P., Wang, J., Liu, J., Liu, Z., Li, D., Fang, Y., Yong, J., Wang, B., et al. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i27.39395 · Cited by 1 · see citing works
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 1 · see citing works
TLR: Codebase-Level C Memory Management Error Repair with Large Language Models
Cheng, X. Y., Guo, Z., Huo, H., & Sui, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797085 · Cited by 1 · see citing works
MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding
Zhang, J. M., Ye, Q., Zhou, H., Liang, H., & Luo, F. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1094 · Cited by 1 · see citing works
Wen, X.-C., Chen, B., Lan, H., Yu, H., Di, P., & Gao, C. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834438 · Cited by 1 · see citing works
Li, Z., Li, Z., Guo, Z., Ren, X., & Huang, C. (2025) · arXiv · doi:10.48550/arxiv.2512.07921
Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Self-citationShen, X., Liu, Y., Dai, Y., Wang, Y., Miao, R., Tan, Y., Pan, S., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2505.23352 · Cited by 6 · see citing works
eHAPPA: Efficient and Scalable Resilience Prediction in HPC Applications with Low-Rank AdaptationPreprint
Jiang, H., Zhu, J., Fang, B., BARKER, K. J., Chen, C., Jin, R., & Guan, Q. (2025) · doi:10.36227/techrxiv.175001095.58343307/v1
Repository-Level Code Generation Method Enhanced by Context-Dependent Graph Retrieval
Zhang, H., Li, A., Kuai, Z., Wei, Z., Kuang, L., & Xia, Y. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93257-1_3
Yuma: Geração de Workflows Multiagentes a partir de Linguagem Natural
Neves, A., Mantovani, D., Campelo, C., Souza, G., Souza, D., Furtado, B., Marinho, L. B., & Araújo, E. (2025) · Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia) · doi:10.5753/webmedia_estendido.2025.16421
Liu, S., & Bi, Y. (2025) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4070577
Improving LLM-Assisted Hydropower Operation Through Data Query Enhanced Agent
Luo, W., Zhang, F., Wang, L., Hu, S., Xu, B., & Zhao, J. (2026) · Lecture notes in civil engineering · doi:10.1007/978-981-95-4889-7_49
Hu, H., Yang, X., Wang, J., Zhang, Q., Tang, M., Song, C., Shang, Y., He, C., & Li, W. (2026) · doi:10.36227/techrxiv.177083699.99572896/v1
Collaborative Multi-Agent Framework for Autonomous Code Generation and Debugging
Nazareth, P., Uppoor, P., N, S. H., R, S. S., & Poojary, Y. (2025) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict67193.2025.11399205
LLM Agent Memory: A Survey from a Unified Representation–Management PerspectivePreprint
Tang, Z., He, X., Zhao, T., Wei, F., Liu, X., Dong, P., (32718), Q. W., Li, Q., Wang, H., Chen, R., et al. (2026) · Preprints.org · doi:10.20944/preprints202603.0359.v1
LLMs Can Code, But Can They Do Designƒ
Guruli, A., Gadelia, M., & Tichy, W. F. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418469
Duc, H., Ngo, M. D., Le, T., Le, D.-K., & Le, M. (2026) · Engineering Research Express · doi:10.1088/2631-8695/ae495d
Failure Localization in Multi-Agent Code Generation via Knowledge-Guided and Transferable Reasoning
Geng, M., Gu, S., Liu, Z., Xu, C., Qu, Z., & Wang, H. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i1.36993
LLM Agent Memory: A Survey from a Unified Representation–Management PerspectivePreprint
Tang, Z., He, X., Zhao, T., Wei, F., Liu, X., Dong, P., Wang, Q., Li, Q., Wang, H., Chen, R., et al. (2026) · Preprints.org · doi:10.20944/preprints202603.0359.v2
LLM-based multi-agent systems for code generation: A multi-vocal literature review
Rasheeda, Z., Waseema, M., Kemella, K.-K., Saari, M., & Abrahamsson, P. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113142
Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
Liu, Z., Huang, Y., Xie, T., (752030), R. Z., Li, W., & Duan, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37314-4_2
Guo, H., Liang, Y.-W., Cheng, X., Ellington, C. N., Xie, P.-T., Song, L., & Xing, E. (2026) · bioRxiv · doi:10.64898/2026.04.20.719735 · Cited by 1 · see citing works
Song, X., Pei, L., Wu, J., Zeng, Y., He, G., Zuo, C., Liu, X., Zhao, Q., Guo, S., & Liu, X. (2026) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2026.240521
- Authors
- Zhang, K., Li, J., Li, G., Shi, X., & Jin, Z.
- Affiliations
- Peking University (CN)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 13643-13658
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671013
- Cited by
- 138
- References
- 24
- Citations by year
- 2026: 81 · 2025: 47 · 2024: 10
- FWCI
- 44.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61832009, 62192730, 62072007, 62192731, 62192733)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code generation, large language models, LLM agents, tool integration, software repositories, code testing, information retrieval, agent strategies, benchmark, HumanEval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
