CoChat
Sign Up
PreprintCited by 3Software Engineering ResearchSoftware Engineering Techniques and PracticesTopic Modeling

CodeS: Natural Language to Code Repository via Multi-Layer Sketch

Open access
Zan, Daoguang, Yu, Ailun, Liu, Wei, Chen, Dong, Shen, Bo, Li, Wei, Yao, Yafen, Gong, Yongshun +6 more·2024·arXiv
DOI 10.48550/arxiv.2403.16443arXiv 2403.16443
Open this paper in CoChat — free accountOpen PDF

Abstract

The impressive performance of large language models (LLMs) on code-related tasks has shown the potential of fully automated software development. In light of this, we introduce a new software engineering task, namely Natural Language to code Repository (NL2Repo). This task aims to generate an entire code repository from its natural language requirements. To address this task, we propose a simple yet effective framework CodeS, which decomposes NL2Repo into multiple sub-tasks by a multi-layer sketch. Specifically, CodeS includes three modules: RepoSketcher, FileSketcher, and SketchFiller. RepoSketcher first generates a repository's directory structure for given requirements; FileSketcher then generates a file sketch for each file in the generated structure; SketchFiller finally fills in the details for each function in the generated file sketch. To rigorously assess CodeS on the NL2Repo task, we carry out evaluations through both automated benchmarking and manual feedback analysis. For benchmark-based evaluation, we craft a repository-oriented benchmark, SketchEval, and design an evaluation metric, SketchBLEU. For feedback-based evaluation, we develop a VSCode plugin for CodeS and engage 30 participants in conducting empirical studies. Extensive experiments prove the effectiveness and practicality of CodeS on the NL2Repo task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.16443

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 63

  1. Zan D, Chen B, Zhang F, et al. Large language models meet nl2code: A survey[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023: 7443-7464.
  2. Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  3. Li Y, Choi D, Chung J, et al. Competition-level code generation with alphacode[J]. Science, 2022, 378(6624): 1092-1097.
  4. Nijkamp E, Pang B, Hayashi H, et al. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022.
  5. Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you![J]. arXiv preprint arXiv:2305.06161, 2023. · arXiv
  6. Lozhkov A, Li R, Allal L B, et al. StarCoder 2 and The Stack v2: The Next Generation[J]. arXiv preprint arXiv:2402.19173, 2024. · arXiv
  7. Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023. · arXiv
  8. Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024. · arXiv
  9. Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  10. Ridnik T, Kredo D, Friedman I. Code Generation with Alpha- Codium: From Prompt Engineering to Flow Engineering. arXiv preprint arXiv:2401.08500. 2024 Jan 16. · arXiv
  11. Liu T, Jiang Y E, Monath N, et al. Autoregressive Structured Prediction with Language Models[C]//Findings of the Association for Computa- tional Linguistics: EMNLP 2022. 2022: 993-1005.
  12. Wells C. A generalization of the concept of sketch[J]. Theoretical Computer Science, 1990, 70(1): 159-178.
  13. Tovey M, Porter S, Newman R. Sketching, concept development and automotive design[J]. Design studies, 2003, 24(2): 135-153.
  14. Zan D, Chen B, Yang D, et al. CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation[J]. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022: 2369-2375.
  15. Ellis K, Wong C, Nye M, Sablé-Meyer M, Morales L, Hewitt L, Cary L, Solar-Lezama A, Tenenbaum JB. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. InProceedings of the 42nd acm sigplan international conference on programming language design and implementation 2021 Jun 19 (pp. 835-850).
  16. Levinson R. A general programming language for unified planning and control[J]. Artificial Intelligence, 1995, 76(1-2): 319-375.
  17. Mou L, Li G, Zhang L, et al. Convolutional neural networks over tree structures for programming language processing[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).
  18. Neelakantan A, Le Q V, Abadi M, et al. Learning a Natural Lan- guage Interface with Neural Programmer[C]//International Conference on Learning Representations. 2016.
  19. Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
  20. Lu S, Guo D, Ren S, et al. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation[C]//Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021.
  21. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865.
  22. Qian C, Cong X, Yang C, et al. Communicative agents for software development[J]. arXiv preprint arXiv:2307.07924, 2023. · arXiv
  23. Yang H, Yue S, He Y. Auto-gpt for online decision making: Benchmarks and additional opinions[J]. arXiv preprint arXiv:2306.02224, 2023. · arXiv
  24. Davis A M, Bersoff E H, Comer E R. A strategy for comparing alter- native software development life cycle models[J]. IEEE Transactions on software Engineering, 1988, 14(10): 1453-1461.
  25. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '23). Association for Computing Machinery, New York, NY, USA, 5673-5684.
  26. Fried D, Aghajanyan A, Lin J, et al. InCoder: A Generative Model for Code Infilling and Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022.
  27. Wang Y, Wang W, Joty S, et al. CodeT5: Identifier-aware Unified Pre- trained Encoder-Decoder Models for Code Understanding and Genera- tion[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 8696-8708.
  28. Christopoulou F, Lampouras G, Gritta M, et al. Pangu-coder: Pro- gram synthesis with function-level language modeling[J]. arXiv preprint arXiv:2207.11280, 2022. · arXiv
  29. Dou S, Liu Y, Jia H, et al. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback[J]. arXiv preprint arXiv:2402.01391, 2024. · arXiv
  30. Shen B, Zhang J, Chen T, et al. Pangu-coder2: Boosting large language models for code with ranking feedback[J]. arXiv preprint arXiv:2307.14936, 2023. · arXiv
  31. Zheng W, Sharan SP, Jaiswal AK, Wang K, Xi Y, Xu D, Wang Z. Outline, then details: Syntactically guided coarse-to-fine code genera- tion. InInternational Conference on Machine Learning 2023 Jul 3 (pp. 42403-42419). PMLR.
  32. Zelikman E, Huang Q, Poesia G, Goodman N, Haber N. Parsel: Algorith- mic Reasoning with Language Models by Composing Decompositions. Advances in Neural Information Processing Systems. 2024 Feb 13;36.
  33. Nguyen N, Nadi S. An empirical evaluation of GitHub copilot's code suggestions[C]//Proceedings of the 19th International Conference on Mining Software Repositories. 2022: 1-5.
  34. Dakhel A M, Majdinasab V, Nikanjam A, et al. Github copilot ai pair programmer: Asset or liability?[J]. Journal of Systems and Software, 2023, 203: 111734.
  35. Yu H, Shen B, Ran D, Zhang J, Zhang Q, Ma Y, Liang G, Li Y, Wang Q, Xie T. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering 2024 Feb 6 (pp. 1- 12).
  36. Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations 2023 Oct 13.
  37. Iyer S, Konstas I, Cheung A, et al. Mapping Language to Code in Programmatic Context[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1643-1652.
  38. Yin P, Deng B, Chen E, et al. Learning to mine aligned code and natural language pairs from stack overflow[C]//Proceedings of the 15th international conference on mining software repositories. 2018: 476-486.
  39. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865.
  40. Ding Y, Wang Z, Ahmad W, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion[J]. Advances in Neural Information Processing Systems, 2024, 36.
  41. Liu T, Xu C, McAuley J. RepoBench: Benchmarking Repository- Level Code Auto-Completion Systems[C]//The Twelfth International Conference on Learning Representations. 2023.
  42. Zhang F, Chen B, Zhang Y, et al. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 2471-2484.
  43. Oda Y, Fudaba H, Neubig G, et al. Learning to generate pseudo-code from source code using statistical machine translation[C]//2015 30th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2015: 574-584.
  44. Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint arXiv:2108.07732, 2021. · arXiv
  45. Hendrycks D, Basart S, Kadavath S, et al. Measuring Coding Challenge Competence With APPS[C]//Thirty-fifth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
  46. Cassano F, Gouwar J, Nguyen D, et al. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation[J]. IEEE Transactions on Software Engineering, 2023.
  47. Lai Y, Li C, Wang Y, et al. DS-1000: A natural and reliable bench- mark for data science code generation[C]//International Conference on Machine Learning. PMLR, 2023: 18319-18345.
  48. Ren S, Guo D, Lu S, et al. Codebleu: a method for automatic evaluation of code synthesis[J]. arXiv preprint arXiv:2009.10297, 2020. · arXiv
  49. Papineni K, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002: 311- 318.
  50. Wolf T, Debut L, Sanh V, et al. Huggingface's transformers: State-of- the-art natural language processing[J]. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  51. Loshchilov I, Hutter F. Decoupled Weight Decay Regulariza- tion[C]//International Conference on Learning Representations. 2018.
  52. Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with pagedattention[C]//Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.
  53. Su J, Ahmed M, Lu Y, et al. Roformer: Enhanced transformer with rotary position embedding[J]. Neurocomputing[J]. Advances in Neural Information Processing Systems, 2024, 568: 127063.
  54. Yen H, Gao T, Chen D. Long-Context Language Modeling with Parallel Context Encoding. arXiv preprint arXiv:2402.16617. 2024 Feb 26. · arXiv
  55. Chen Y, Qian S, Tang H, et al. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models[C]//The Twelfth International Conference on Learning Representations. 2023.
  56. Zhang P, Liu Z, Xiao S, Shao N, Ye Q, Dou Z. Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon. arXiv preprint arXiv:2401.03462. 2024 Jan 7. · arXiv
  57. Guo D, Ren S, Lu S, et al. GraphCodeBERT: Pre-training Code Rep- resentations with Data Flow[C]//International Conference on Learning Representations. 2020.
  58. Huang D, Nan Z, Hu X, Jin P, Peng S, Wen Y, Zhang R, Du Z, Guo Q, Pu Y, Chen Y. ANPL: Towards Natural Programming with Interactive Decomposition. Advances in Neural Information Processing Systems. 2024 Feb 13;36.
  59. Li J, Li Y, Li G, et al. Skcoder: A sketch-based approach for automatic code generation[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023: 2124-2135.
  60. Ge S, Goswami V, Zitnick C L, et al. Creative sketch generation[J]. arXiv preprint arXiv:2011.10039, 2020. · arXiv
  61. Lozhkov A, Li R, Allal LB, Cassano F, Lamy-Poirier J, Tazi N, Tang A, Pykhtar D, Liu J, Wei Y, Liu T. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173. 2024 Feb 29. · arXiv
  62. Zhang B, Liu Z, Cherry C, et al. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method[C]//The Twelfth International Conference on Learning Representations. 2023.
  63. Hall M W, Kennedy K. Efficient call graph analysis[J]. ACM Letters on Programming Languages and Systems (LOPLAS), 1992, 1(3): 227-242.

Cited by

Sorted by citations

Authors
Zan, D., Yu, A., Liu, W., Chen, D., Shen, B., Li, W., Yao, Y., Gong, Y., Chen, X., Guan, B., Yang, Z., Wang, Y., Wang, Q., & Cui, L.
Affiliations
  • Chinese Academy of Sciences (CN)
  • Shandong University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.16443
arXiv
2403.16443
OpenAlex
W4393213295
Cited by
2
References
0
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Key Research and Development Program of China (2022ZD0120201)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, user feedback evaluation, software development automation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review