CoChat
Sign Up
PreprintCited by 17Software Engineering ResearchTeaching and Learning ProgrammingSoftware Testing and Debugging Techniques

CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging

Open access
Islam, Md. Ashraful, Ali, Mohammed Eunus, Parvez, Md Rizwan·2025·arXiv
DOI 10.48550/arxiv.2502.05664arXiv 2502.05664
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have made significant strides in code generation and problem solving. Current approaches employ external tool-based iterative debuggers that use compiler or other tool-based runtime feedback to refine coarse programs generated by various methods. However, the effectiveness of these approaches heavily relies on the quality of the initial code generation, which remains an open challenge. In this paper, we introduce CodeSim, a novel multi-agent code generation framework that comprehensively addresses the stages of program synthesis-planning, coding, and debugging-through a human-like perception approach. As human verifies their understanding of any algorithms through visual simulation, CodeSim uniquely features a method of plan verification and internal debugging through the step-by-step simulation of input/output. Extensive experiments across seven challenging competitive problem-solving and program synthesis benchmarks demonstrate CodeSim's remarkable code generation capabilities. Our framework achieves new state-of-the-art (pass@1) results-(HumanEval 95.1%, MBPP 90.7%, APPS 22%, and CodeContests 29.1%). Furthermore, our method shows potential for even greater enhancement when cascaded with external debuggers. To facilitate further research and development in this area, we have open-sourced our framework in this link (https://kagnlp.github.io/codesim.github.io/).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.05664

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other recent multi-agent coding systems [Pan et al., 2026, Islam et al., 2025, Peng et al., 2026] share the LLM-team-coding goal but use sequential or staged-role paradigms.

    “Other recent multi-agent coding systems [Pan et al., 2026, Islam et al., 2025, Peng et al., 2026] share the LLM-team-coding goal but use sequential or staged-role paradigms.”

    From AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace (2026) · cites this paper as Islam et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Self-collaboration code generation via ChatGPT

    Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · cited 3× · see citing works

    “…recent paradigms have shifted toward plan-based (Jiang et al., 2023b), sampling or tree-searching (Zhou et al., 2023), self-retrieval (Yasunaga et al., 2023), and diverse agent-based approaches (Zhang et al., 2024; Qian et al., 2024; Shinn et al., 2023; Huang et al., 2023; Dong et al., 2023b).”

Flagged first, then by citations

Research works printed but not linked: 55

  1. Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333. · arXiv
  2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural lan- guages. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 1536-1547.
  3. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999. · arXiv
  4. Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46(1):317-330.
  5. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196. · arXiv
  6. Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for model- ing source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineer- ing, ESEC/FSE 2017, pages 763-773, New York, NY, USA. ACM. · DOI · Paper page
  7. Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. · arXiv
  8. Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software. Commun. ACM, 59(5):122-131. · DOI · Paper page
  9. Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent- based code generation with iterative testing and opti- misation. arXiv preprint arXiv:2312.13010. · arXiv
  10. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Day- iheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. · arXiv
  11. Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024a. MapCoder: Multi-agent code generation for competitive problem solving. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4912-4944, Bangkok, Thailand. Association for Computational Linguistics. · DOI · Paper page
  12. Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, and Md Rizwan Parvez. 2024b. Open-RAG: Enhanced retrieval aug- mented reasoning with open-source large language models. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024, pages 14231- 14244, Miami, Florida, USA. Association for Com- putational Linguistics. · DOI
  13. Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023b. Self-planning code genera- tion with large language model. arXiv preprint arXiv:2303.06689. · arXiv
  14. Dongming Jin, Zhi Jin, Xiaohong Chen, and Chun- hui Wang. 2024a. Mare: Multi-agents collabora- tion framework for requirements engineering. arXiv preprint arXiv:2405.03256. · arXiv
  15. Haolin Jin, Zechao Sun, Yiheng Yang, and Huaming Chen. 2024b. Rgd: Multi-llm based agent debug- ger via refinement and generation guidance. arXiv preprint arXiv:2410.01242. · arXiv
  16. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. xcodeeval: A large scale multi- lingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004. · arXiv
  17. Uirá Kulesza, Alessandro Garcia, Carlos Lucena, and Paulo Alencar. 2004. A generative approach for multi-agent system development. In International Workshop on Software Engineering for Large-Scale Multi-agent Systems, pages 52-69. Springer.
  18. Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Sai- ful Bari, Mizanur Rahman, Mohammad Abdul- lah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. 2024. A sys- tematic survey and critical review on evaluating large language models: Challenges, limitations, and recom- mendations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 13785-13816, Miami, Florida, USA. As- sociation for Computational Linguistics. · DOI
  19. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328.
  20. Kyla Levin, Nicolas van Kempen, Emery D Berger, and Stephen N Freund. 2024. Chatdbg: An ai-powered debugging assistant. arXiv preprint arXiv:2403.16354. · arXiv
  21. Jingyao Li, Pengguang Chen, and Jiaya Jia. 2023. Mot- coder: Elevating large language models with modular of thought for challenging programming tasks. arXiv preprint arXiv:2312.15960. · arXiv
  22. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022a. Competition-level code generation with alphacode. Science, 378(6624):1092-1097.
  23. Yujia Li, David Choi, Junyoung Chung, Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d'Autume, Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022b. Competition-level code generation with al- phacode. Science, 378(6624):1092-1097. · DOI · Paper page
  24. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. 2023. Is your code generated by chat- GPT really correct? rigorous evaluation of large lan- guage models for code generation. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
  25. Zohar Manna and Richard J. Waldinger. 1971. To- ward automatic program synthesis. Commun. ACM, 14(3):151-165. · DOI · Paper page
  26. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. · arXiv
  27. Emilio Parisotto and Ruslan Salakhutdinov. 2017. Neu- ral map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1702.08360. · arXiv
  28. Md Rizwan Parvez. 2024. Evidence to generate (e2g): A single-agent two-step prompting for context grounded and retrieval augmented reasoning. arXiv preprint arXiv:2401.05787. · arXiv
  29. Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and sum- marization. arXiv preprint arXiv:2108.11601. · arXiv
  30. Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language mod- els for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2373-2383, Melbourne, Australia. Association for Computational Linguistics. · DOI
  31. Md Rizwan Parvez and Kai-Wei Chang. 2021. Evalu- ating the values of sources in transfer learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5084-5116, Online. Association for Computa- tional Linguistics. · DOI
  32. Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2023. Retrieval enhanced data augmentation for question answer- ing on privacy policies. In Proceedings of the 17th Conference of the European Chapter of the Associa- tion for Computational Linguistics, pages 201-210, Dubrovnik, Croatia. Association for Computational Linguistics. · DOI
  33. Huy Nhat Phan, Phong X Nguyen, and Nghi DQ Bui. 2024. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409.16299. · arXiv
  34. Oleksandr Polozov and Sumit Gulwani. 2015. Flash- meta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN Interna- tional Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 107- 126.
  35. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174-15186, Bangkok, Thailand. Association for Computational Linguistics. · DOI · Paper page
  36. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. CoRR, abs/1704.07535. · arXiv
  37. Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500. · arXiv
  38. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. · arXiv
  39. Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang, Rakesh Menon, Md Parvez, and Zhe Feng. 2023. Delu- cionQA: Detecting hallucinations in domain-specific question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 822-835, Singapore. Association for Computational Linguistics. · DOI
  40. Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From code to correctness: Clos- ing the last mile of code generation with hierarchical debugging. arXiv preprint arXiv:2410.01215. · arXiv
  41. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Re- flexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  42. Kashun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with chain-of-thought from labeled data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113-12139, Singapore. Asso- ciation for Computational Linguistics. · DOI
  43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  44. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code un- derstanding and generation. arXiv preprint arXiv:2109.00859. · arXiv
  45. Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023b. Learning to fil- ter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377. · arXiv
  46. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022a. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  48. Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H Chi, and Denny Zhou. 2023. Large language models as ana- logical reasoners. arXiv preprint arXiv:2310.01714. · arXiv
  49. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. CoRR, abs/1704.01696. · arXiv
  50. Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019. CoSQL: A conversational text-to-SQL challenge towards cross- domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 1962- 1979, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
  51. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643- 13658, Bangkok, Thailand. Association for Compu- tational Linguistics. · DOI · Paper page
  52. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493. · arXiv
  53. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. · arXiv
  54. Li Zhong, Zilong Wang, and Jingbo Shang. 2024. De- bug like a human: A large language model debugger via verifying runtime execution step by step. In Find- ings of the Association for Computational Linguis- tics: ACL 2024, pages 851-870, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  55. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023. Lan- guage agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. · arXiv

Cited by

Sorted by citations

Authors
Islam, M. A., Ali, M. E., & Parvez, M. R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
5113-5139
DOI
10.48550/arxiv.2502.05664
arXiv
2502.05664
OpenAlex
W4411120270
Cited by
17
References
1
Citations by year
2026: 13 · 2025: 3 · 2024: 1
FWCI
18.30
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
program synthesis, simulation-based planning, debugging, large language models, plan verification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review