InCoder: A Generative Model for Code Infilling and Synthesis
Open accessAbstract
Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
They lack many models capable of generating code.
“However, we lack many models capable of generating code (Black et al., 2021;Fried et al., 2022;Black et al., 2022;Wang & Komatsuzaki, 2021;Biderman et al., 2023b).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Fried et al., 2022
The Related Work section (p. 9) describes a “flurry of recent work” on code language models and identifies models that generate programs. InCoder itself is a code-generating model (Abstract, p. 1), but the paper does not support the cited sentence’s claim that such models are scarce. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe study used Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.
“Following existing works evaluating LLM code generation [12, 23] and common evaluation practices over LiveCodeBench and DevEval, we use Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.”
From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as [23]
Section C.6 (p. 20) confirms that InCoder used pass@1 to evaluate code synthesis on HumanEval and MBPP, supporting the citation as a precedent for that metric. It does not say pass@1 is more appropriate than code-similarity scores: it also reports pass@10 and pass@100 for HumanEval, while Table 1 (p. 5) reports exact match alongside pass rate. The citing authors’ choice to use pass@1 for all their research questions is their own methodology, not a finding attributed to InCoder. (LLM, conf=0.87)
Experimental — accuracy not yet measuredLonger contexts enable moving forward to smart context composition techniques, including fill-in-the-middle and retrieval-augmented generation methods.
“Longer contexts enable us to move forward to”
From Context Composing for Full Line Code Completion (2024) · cites this paper as [3]
Section 2.2 (p. 3) describes code infilling using left and right context, and Section 4.1 (p. 5) reports larger gains as more right-sided context becomes available. Section 7 (p. 9) identifies a similar fill-in-the-middle objective. The paper does not establish that longer context windows enable these techniques, and the passages found do not address retrieval-augmented generation. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPretrained large language models have demonstrated strong capabilities at solving a wide range of programming tasks.
“Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks.”
From Learning Performance-Improving Code Edits (2023) · cites this paper as Fried et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundMethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 2× · see citing works
“For our learning rate scheduler, we use the built-in polynomial decay learning rate scheduler available in [49] with 1500 warmup updates.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Neural models for code generation have either utilized a left-to-right (causal) autoregressive language modeling objective [16, 18] or, as BERT does, a masked language modeling objective [22, 24].”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · see citing works
Self-citationSpace/time trade-offs in hash coding with allowable errors
Bloom, B. (1970) · CACM · doi:10.1145/362686.362692 · Cited by 8,424 · cited 1× · see citing works
“(6)We implement match checking using a Bloom filter [15] whose keys are: the file extension, number of tokens in the file, and an MD5 hash [53] of the sequence of tokens, which is highly accurate at identifying files with exactly matching token sequences.”
Methodfairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · cited 4× · see citing works
“We use the same architecture as the dense 6.7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
MethodInfluentialTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent [76, 65, 48].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5× · see citing works
“[21] performed best, we found that our model did not benefit from additional examples in the prompt, which we attribute to much smaller size of our model (6.”
BackgroundMethodInfluentialScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps [39, 32].”
BackgroundWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1× · see citing works
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · ESEC/FSE '11 · doi:10.1145/2025113.2025179 · Cited by 1,445 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 10× · see citing works
“Neural models for code generation have either utilized a left-to-right (causal) autoregressive language modeling objective [16, 18] or, as BERT does, a masked language modeling objective [22, 24].”
BackgroundMethodInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 1× · see citing works
“For all three inference methods, we obtain generations from the model using Fairseq’s implementation [47] of top-p (nucleus) sampling [33] with p = 0.”
MethodFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1× · see citing works
“Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent [76, 65, 48].”
BackgroundCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 2× · see citing works
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
MethodAutomating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/1926385.1926423 · Cited by 1,039 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundUnified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,000 · cited 2× · see citing works
“, [18, 45, 4]), we evaluate our model zero-shot, with no explicit documentation string supervision.”
MethodNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1× · see citing works
“To increase the amount of context that our code model can condition on, the length of documents that the model can generate, and the efficiency of training and inference, we train a byte-level BPE tokenizer [56, 51].”
A systematic evaluation of large language models of codePreprint
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · MAPS@PLDI · doi:10.48550/arxiv.2202.13169 · Cited by 925 · cited 3× · see citing works
“Large language models trained on vast repositories of code have demonstrated remarkable progress in neural program synthesis and related tasks [18, 10, 68, 46, 21].”
BackgroundMethodCombinatorial sketching for finite programs
Solar-Lezama, A., Tancau, L., Bodík, R., Seshia, S., & Saraswat, V. (2006) · ASPLOS XII · doi:10.1145/1168857.1168907 · Cited by 922 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1× · see citing works
“0 — Unreleased LaMDA [10, 60, 21] 137 None None ??? — 14.”
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1× · see citing works
“We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
MethodGulwani, S., Polozov, O., & Singh, R. (2017) · Software Systems Safety · doi:10.3233/978-1-61499-385-8-43 · Cited by 748 · cited 1×
“Machine Learning for Program Synthesis Program synthesis approaches directly generate programs from a specification of functionality [29].”
MethodDeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundSequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z.-M., Kommrusch, S., Tufano, M., Pouchet, L., Poshyvanyk, D., & Martin, M. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 585 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodDefending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · cited 1× · see citing works
“We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
MethodCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 11× · see citing works
“We also include the performance of the supervised baseline as reported in the CodeXGLUE paper.”
MethodInfluentialThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 2× · see citing works
“The Pile [26], which was used to train recent generative code models that we compare to in Section 6, also contains these questions and answers but does not contain comments.”
MethodCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 4× · see citing works
“CodeXGLUE cloze is created from CodeSearchNet to evaluate CodeBERT and consists of a short natural language description followed by code in several programming languages.”
MethodInfluentialMask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464 · cited 1× · see citing works
“Finally, our model lays a foundation for future work on supervised infilling & editing via model fine-tuning, as well as performing iterative decoding, where the model can be used to refine its own output [27].”
MethodSelf-citationDeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 8× · see citing works
“We compare model pass@1 scores on the HumanEval [18] and MBPP [10] left-to-right synthesis benchmarks, with results in Table 7.”
BackgroundMethodInfluentialCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 5× · see citing works
“While differences in details of the Transformer model architectures, datasets, and training procedures across papers and experimental setups make a rigorous comparison impossible, we note that our model achieves roughly comparable performance on the HumanEval metrics to CodeGen-Multi [46], which is also a ∼6B paramete…”
BackgroundMethodInfluentialInsertion Transformer: Flexible Sequence Generation via Insertion OperationsPreprint
Stern, M., Chan, W., Kiros, J., & Uszkoreit, J. (2019) · International Conference on Machine Learning · doi:10.48550/arxiv.1902.03249 · Cited by 300 · cited 1× · see citing works
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundThe adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · cited 1× · see citing works
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
BackgroundEfficient Large Scale Language Modeling with Mixtures of ExpertsPreprint
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J.-F., Iyer, S., Pasunuru, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2112.10684 · Cited by 262 · cited 1× · see citing works
“7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
BackgroundSelf-citationCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · cited 2× · see citing works
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
MethodHellendoorn, V. J., Bird, C., Barr, E. T., & Allamanis, M. (2018) · ESEC/SIGSOFT FSE · doi:10.1145/3236024.3236051 · Cited by 227 · cited 2× · see citing works
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodEnabling Language Models to Fill in the Blanks
Donahue, C., Lee, M., & Liang, P. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2005.05339 · Cited by 222 · cited 1×
“[23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al.”
BackgroundLearning to mine aligned code and natural language pairs from stack overflow
Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196408 · Cited by 218 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundPyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.03150 · Cited by 168
Allamanis, M., Barr, E. T., Ducousso, S., & Gao, Z. (2020) · ACM-SIGPLAN Symposium on Programming Language Design and Implementation · doi:10.1145/3385412.3385997 · Cited by 156 · cited 1× · see citing works
“Some recent work has developed supervised machine learning approaches for predicting type annotations for dynamically-typed languages including Python [69, 6, 50] and TypeScript [31, 66, 36].”
MethodScaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · cited 1× · see citing works
“Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps [39, 32].”
BackgroundTypeWriter: neural type prediction with search-based validation
Pradel, M., Gousios, G., Liu, J., & Chandra, S. (2019) · ESEC/SIGSOFT FSE · doi:10.1145/3368089.3409715 · Cited by 141 · cited 9× · see citing works
“We compare our zero-shot model to one such approach for Python, TypeWriter [50], which combines a neural architecture for type hint prediction with a search-based incremental type validation procedure.”
BackgroundMethodInfluentialPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 123 · see citing works
LambdaNet: Probabilistic Type Inference using Graph Neural NetworksPreprint
Wei, J., Goyal, M., Durrett, G., & Dillig, I. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2005.02161 · Cited by 120 · cited 3× · see citing works
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodAdapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 118 · see citing works
Self-citationPython probabilistic type inference with natural language support
Xu, Z.-G., Zhang, X., Chen, L., Pei, K.-X., & Xu, B.-W. (2016) · SIGSOFT FSE · doi:10.1145/2950290.2950343 · Cited by 108 · cited 1× · see citing works
“Some recent work has developed supervised machine learning approaches for predicting type annotations for dynamically-typed languages including Python [69, 6, 50] and TypeScript [31, 66, 36].”
MethodSPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundDOBF: A Deobfuscation Pre-Training Objective for Programming LanguagesPreprint
Rozière, B., Lachaux, M.-A., Szafraniec, M., & Lample, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07492 · Cited by 96 · cited 2× · see citing works
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundRestoring ancient text using deep learning: a case study on Greek epigraphy
Assael, Y., Sommerschield, T., & Prag, J. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1668 · Cited by 88 · cited 1× · see citing works
“, editing sentences [57], restoring ancient text [9], and fixing bugs in source code.”
BackgroundCodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86 · cited 1× · see citing works
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
CLN2INV: Learning Loop Invariants with Continuous Logic NetworksPreprint
Ryan, G., Wong, J., Yao, J., Gu, R.-H., & Jana, S. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1909.11542 · Cited by 82 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodGraph-based, Self-Supervised Program Repair from Diagnostic Feedback
Yasunaga, M., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10636 · Cited by 81 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodAutoPandas: neural-backed generators for program synthesis
Bavishi, R., Lemieux, C., Fox, R., Sen, K., & Stoica, I. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360594 · Cited by 73 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundDeduplicating Training Data Mitigates Privacy Risks in Language Models
Kandpal, N., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2202.06539 · Cited by 68 · cited 1× · see citing works
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
BackgroundSelf-citationKERMIT: Generative Insertion-Based Modeling for SequencesPreprint
Chan, W., Kitaev, N., Guu, K., Stern, M., & Uszkoreit, J. (2019) · arXiv.org · doi:10.48550/arxiv.1906.01604 · Cited by 68 · cited 1× · see citing works
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · see citing works
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
Agashe, R., Iyer, S. V., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1546 · Cited by 48 · cited 2× · see citing works
“We additionally removed repositories in the evaluation sets of the JuICe tasks [1], although we did not evaluate our model on these tasks in this present work.”
BackgroundMethodSelf-citationEfficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47 · cited 1× · see citing works
“In addition, concurrent to our work, OpenAI developed a fill-in-the-middle (FIM) training objective similar to the causal masking objective we use, trained code models with it, and evaluated on the HumanEval infilling tasks we introduce here (Bavarian et al., 2022).”
MethodBreak-It-Fix-It: Unsupervised Learning for Program RepairPreprint
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 43 · cited 1× · see citing works
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 6× · see citing works
“Most related to our approach is the work of Donahue et al. [23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al. [7], who train an infilling-capable, AST-structured generative model of code on a smaller scale.”
BackgroundMethodInfluentialSelf-citationLearning type annotation: is big data enough?
Jesse, K., Dévanbu, P., & Ahmed, T. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473135 · Cited by 41 · cited 2× · see citing works
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodHTLM: Hyper-Text Pre-Training and Prompting of Language Models
Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2107.06955 · Cited by 35 · cited 1× · see citing works
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundSelf-citationSpreadsheetCoder: Formula Prediction from Semi-structured ContextPreprint
Chen, X., Maniatis, P., Singh, R., Sutton, C. A., Dai, H., Lin, M., & Zhou, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15339 · Cited by 18 · cited 1× · see citing works
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2107.06499 · Cited by 11 · cited 1× · see citing works
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
BackgroundReflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models
West, P., Lu, X., Holtzman, A., Bhagavatula, C., Hwang, J. D., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.114 · Cited by 9 · cited 1× · see citing works
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundXL-Editor: Post-editing Sentences with XLNetPreprint
Shih, Y.-S., Chang, W.-C., & Yang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10479 · Cited by 7 · cited 1× · see citing works
“, editing sentences [57], restoring ancient text [9], and fixing bugs in source code.”
Background
Flagged first, then by citations
9 of the 81 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In ICML , pp. 245-256. PMLR, 2020.
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In ICML , 2020.
- Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Transformers . O'Reilly Media, Inc., 2022.
- John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In AAAI , 1996.
Cited by
Sorted by citations
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · see citing works
“Our code-training for 7B, 13B and 70B Code Llama models features a multitask objective (Fried et al., 2023) consisting of both autoregressive and causal infilling prediction, enabling applications such as real-time completion in source code editors or docstring generation.”
BackgroundMethodInfluentialLarge language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“For example, Murali et al. [37] reported the experience of deploying CodeCompose, a code completion tool based on the Incoder LLM [38], at Meta.”
MethodMathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544 · see citing works
“For example, all experiments on admissible sets improve on the previous best capacity lower bound, with 60% of experiments on (12,7) I finding a full-size admissible set.”
BackgroundAutomated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433 · see citing works
“• INCODER [33] A model designed for code infilling by adopting a causal masking objective [45].”
InfluentialGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · see citing works
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · see citing works
“This led to the development of several code generation models such as Codex [9], AlphaCode [22], Google’s program synthesis model [23], PolyCoder [24], InCoder [25] and CodeGen [26].”
MethodDeng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326 · see citing works
“95 following previous studies [12, 21], and <0G_C>:4=B=256, C4<?4A0CDA4=0.”
BackgroundMethodResultInfluentialCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · see citing works
“Second, in addition to code generation and code completion as Codex and others, CodeGeeX supports the tasks of code explanation and code translation between language pairs (Cf. Figure 1 (a)).”
BackgroundMethodInfluentialSelf-citationSelf-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · see citing works
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259 · see citing works
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240 · see citing works
“[9] introduce a large language model, InCoder, and try zero-shot training on CodeXGLUE Python dataset.”
MethodMetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214 · see citing works
“We conduct a comparative evaluation by comparing recent domain-specific LLMs in the code generation field, including AlphaCode [28], Incoder [43], CodeGeeX [44], and CodeX [24] and CodeT [45], with general domain LLMs such as PaLM [46], GPT-4 [47], and MetaGPT.”
MethodImpact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202 · see citing works
“In this work, we evaluate ten code language models of four types (PLBART [23], CodeT5 [24], CodeGen [35], and InCoder [45]) on four benchmarks (Defects4J v1.”
MethodInferFix: End-to-End Program Repair with LLMs
Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., & Svyatkovskiy, A. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613892 · Cited by 191 · see citing works
“[12] permute ordering of the spans in the original prompt to train the model to infill.”
BackgroundRetrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178 · see citing works
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149 · see citing works
“After the success of CodeX, various models such as InCoder [13], Google Alphacode [23], and Amazon CodeWhisperer [1] have been emerging, resulting in remarkable improvements in the effectiveness of code generation.”
MethodEvolutionary Computation in the Era of Large Language Model: Survey and Roadmap
Wu, X., Wu, S.-H., Wu, J., Feng, L., & Tan, K. C. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3506731 · Cited by 177 · see citing works
“It utilizes the generative ability of Codex [109] for seed generation and the infilling ability of InCoder [140] for mutation generation, to automatically produce a large number of input programs that meet the requirements for testing.”
Large Language Models for Code: Security Hardening and Adversarial Testing
He, J., & Vechev, M. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3623175 · Cited by 157 · see citing works
“For instance, the user can use SVENsec to harden open-source LMs [35, 57, 69].”
BackgroundMethodInfluentialLarge Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144 · see citing works
“Models such as Codex [14] and InCoder [48] can perform the opposite direction as well: generate natural language docstring from code snippets alone.”
MethodBorn, J., & Manica, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00639-z · Cited by 151 · see citing works
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 137 · see citing works
“InCoder [40] 2022 670K Python, JavaScript, HTML and 24 more Code Generation, Code Summarization https: //github.”
MethodLLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120 · see citing works
“2b: assert find_First_Missing([1,2,4,6]) ==0 .”
BackgroundDibia, V. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.11 · Cited by 125 · see citing works
Language Models of Code are Few-Shot Commonsense Learners
Madaan, A., Zhou, S., Alon, U., Yang, Y., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.90 · Cited by 131 · see citing works
Lu, J., Yu, L., Xiao-jia, L., Li, Y., & Zuo, C. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00026 · Cited by 133 · see citing works
“are primarily pre-trained on a code-based corpus and they have achieved impressive results in code generation [39]–[42].”
BackgroundLarge Language Models Meet NL2Code: A Survey
Zan, D.-G., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Wang, Y.-J., & Lou, J.-G. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.411 · Cited by 293 · see citing works
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 109 · see citing works
“Since 2020, several code generation models have been trained or fine-tuned on programming language text [1], [3], [4], [6], [16], [17].”
BackgroundMethodInfluentialAssessment of chemistry knowledge in large language models that generate code
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2023) · Digital Discovery · doi:10.1039/d2dd00087c · Cited by 119 · see citing works
LLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M. M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 113 · see citing works
How Effective Are Neural Networks for Fixing Security Vulnerabilities
Wu, Y., Jiang, N., Pham, H. V., Lutellier, T., Davis, J., Tan, L., Babkin, P., & Shah, S. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598135 · Cited by 116 · see citing works
“InCoder [28]: InCoder models follow XGLM [45]’s decoder-only architecture and are pre-trained on the masked span prediction task.”
MethodSiddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101 · see citing works
“InCoder [9] 20 (15.38”
MethodInfluentialWhen Software Security Meets Large Language Models: A Survey
Zhu, X., Zhou, W., Han, Q., Ma, W., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124971 · Cited by 105 · see citing works
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.859 · Cited by 100 · see citing works
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74 · see citing works
“Thus, the LLMs applicable to our framework are limited, i.e., the LLMs without instruction tuning (e.g., InCoder [14] and CodeGen [33]) are not suitable as the base models applied to ClarifyGPT framework.”
Self-citationRefactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 77 · see citing works
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 132 · see citing works
“LLMs have revolutionized various aspects of software engineering [31], [32], including but not limited to automated code generation [2], [33], [34], [35], software testing [36], [37], [38], [39], debugging [40], program repair [41], [42], and document generation [43].”
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 99 · see citing works
Large Language Models and Simple, Stupid Bugs
Jesse, K., Ahmed, T., Dévanbu, P., & Morgan, E. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00082 · Cited by 83 · see citing works
“The capabilities of Codex and similar models [11]–[14] have raised many concerns, and have given rise to different research thrusts.”
BackgroundSource Code Summarization in the Era of Large Language Models
Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., & Chen, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00034 · Cited by 95 · see citing works
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68 · see citing works
Self-Edit: Fault-Aware Code Editor for Code Generation
Zhang, K., Li, Z., Li, J., Li, G., & Jin, Z. G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.45 · Cited by 88 · see citing works
StructCoder: Structure-Aware Transformer for Code Generation
Tipirneni, S., Zhu, M., & Reddy, C. K. A. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3636430 · Cited by 71 · see citing works
“com/microsoft/CodeXGLUE 3We did not include CODEGEN [17] and Incoder [6] in the baselines because these models are trained on much bigger datasets and/or use much larger architectures and it is unfair to compare them with our model and the other baselines used in this paper.”
ResultLanguage Models for Code Completion: A Practical Evaluation
Izadi, M., Katzy, J., van Dam, T., Otten, M., Popescu, R. M., & Arie Van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639138 · Cited by 69 · see citing works
“Thegoalofthisstudyistosystematicallyevaluatethreepub-liccompletionmodels,namelyInCoder[2],UniXcoder[3],and CodeGPT[19]inareal-worldcodingsetup.”
BackgroundMethodInfluentialKNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair
Jiang, N., Lutellier, T., Lou, Y., Tan, L., Goldwasser, D., & Zhang, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00111 · Cited by 61 · see citing works
“A recent direction of DL-based code generation is applying large language models (LLMs) trained on source code to generate code, such as CodeBert [60], CodeT5 [61], CodeGen [62], InCoder [63], and Codex [64].”
MethodSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
“, 2022) or provide cloze-style fill-in blanks (e.g., Lu et al., 2021; Fried et al., 2023), SWE-bench does not provide such explicit guidance.”
BackgroundA systematic literature review on the impact of AI models on the security of code generation
Negri-Ribalta, C., Géraud, R., Sergeeva, A., & Lenzini, G. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1386720 · Cited by 67 · see citing works
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 56 · see citing works
“For example, AlphaCode [2] focuses on solving competitive-level programming problems, while InCoder [100] supports code completion in arbitrary positions using bidirectional contexts.”
RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
Silva, A., Fang, S., & Monperrus, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3581062 · Cited by 49 · see citing works
“This feature is commonly used in LLMs’ training for code [28], [39], [46], [47], and, as such, our approach is easily extensible to these models.”
CAT-LM Training Language Models on Aligned Code And Tests
Rao, N., Jain, K., Alon, U., Goues, C. L., & Hellendoorn, V. J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00193 · Cited by 49 · see citing works
“Tools like Copilot excel at code generation, and can significantly improve the productivity of its users [14].”
AVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.143 · Cited by 46 · see citing works
Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Bhatnagar, A., Jain, S., Beazer, J., Curran, S. C., Hoffnagle, A. M., Ching, K., Martyn, M., Nayfach, S., Ruffolo, J. A., & Madani, A. (2025) · neural information processing systems · doi:10.52202/085713-1536 · Cited by 46 · see citing works
Enhanced automated code vulnerability repair using large language models
de-Fitero-Dominguez, D., Garcia-Lopez, E., Garcia-Cabot, A., & Martinez-Herraiz, J.-J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109291 · Cited by 43 · see citing works
“In addition, we will outline how we evaluate the performance of the fine-tuned models.”
ReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42 · see citing works
Self-citationThe Plastic Surgery Hypothesis in the Era of Large Language Models
Xia, C. S., Ding, Y., & Zhang, L. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00047 · Cited by 36 · see citing works
“Large Language Models (LLMs) trained on large amounts of text combined with code snippets from a broad range of open-source repositories have shown impressive progress in various code-related tasks [44], [45], [46].”
Hajipour, H., Hassler, K., Holz, T., Schönherr, L., & Fritz, M. (2023) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 107 · see citing works
“LLMs have advanced the field of natural language processing in various tasks, including question answering, translation, and reading comprehension [1], [17].”
BackgroundMethodInfluentialHow Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40 · see citing works
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
BackgroundTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40 · see citing works
“Recent advancements have introduced a variety of code large language models (LLMs) [7, 12, 13, 15, 17, 21, 28, 29, 33, 37, 46, 53, 54] constructed upon the Transformer model architecture [45], achieving promising performance in code-related applications [19, 20, 47—50, 52, 57, 59, 60].”
When Language Model Meets Private Library
Zan, D., Chen, B., Lin, Z., Guan, B., Yongji, W., & Lou, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.21 · Cited by 51 · see citing works
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
“In this work, we would like to fill this gap and investigate: How do LLMs perform on To answer this question, we select representative and popular >1B parameters LLMs including InCoder [17], CodeGen [59], Llama 2 [73], and Code Llama [67].”
Multi-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38 · see citing works
“In example 3, given result[char] = 1 in the right context, InCoder model generate result[char]+ = 1.”
MethodInfluentialSelf-citationBias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 38 · see citing works
“In these tasks, ensuring fairness is crucial to maintain ethical standards and often to comply with legal and societal norms.”
CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation
Liu, M., Yang, T., Lou, Y., Du, X., Wang, Y., & Peng, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00159 · Cited by 37 · see citing works
Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models
Zhu, Y., Li, J., Li, G., Zhao, Y., Li, J., Jin, Z. G., & Mei, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27798 · Cited by 48 · see citing works
Generative Type Inference for Python
Peng, Y. F., Wang, C., Wang, W., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00031 · Cited by 36 · see citing works
“Wei et al. [48] propose to use graph neural networks to predict 989 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundMethodWickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 48 · see citing works
Data Formulator: AI-powered Concept-driven Visualization Authoring
Wang, C., Thompson, J., & Lee, B. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326585 · Cited by 35 · see citing works
“Code generation models [7,9,12] and program synthesis techniques [6,14,52,63] enable users to complete tasks without programming by using easier specifications, including natural language, examples, and demonstrations.”
BackgroundMethodKulsum, U., Zhu, H., Xu, B., & d'Amorim, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664770 · Cited by 34 · see citing works
From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining
Ren, X., Ye, X., Zhao, D., Xing, Z., & Yang, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00143 · Cited by 36 · see citing works
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 32 · see citing works
“The recent work, InCoder [18], is adept at handling generative tasks (e.g., comment generation) after learning bidirectional context for infilling arbitrary code lines.”
InfluentialSelf-citationEvaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 33 · see citing works
“The InCoder [29] Forallmodels, we set the sampling temperature to 0.7 (except in Sections VI-D5 and VI-E2 where we explicitly ex-periment with the temperature parameter), and the maximum number of generated tokens to 256.”
MethodInfluentialBigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29 · see citing works
Self-citationLearning-based Widget Matching for Migrating GUI Test Cases
Zhang, Y., Zhang, W., Ran, D., Zhu, Q., Dou, C., Hao, D., Xie, T., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623322 · Cited by 29 · see citing works
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30 · see citing works
“In addition, we also add the FPFT strategy to compare it with PEFT techniques.”
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23 · see citing works
“Represented by Codex [10], CodeGen [23], and others like [24], [25], [26], [27], [28].”
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 34 · see citing works
Fine Tuning Large Language Model for Secure Code Generation
Li, J., Sangalay, A., Cheng, C., Tian, Y., & Yang, J. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652299 · Cited by 28 · see citing works
“CodeBERT (125M) [14], InCoder [15], and Star-Coder [21] are MLMs, which target predicting the masked code according to the code context around the masked pieces.”
BackgroundChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance
Hayet, I., Scott, A., & d’Amorim, M. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519159 · Cited by 27 · see citing works
“A large language model (LLM), such as InCoder [24], Poly-coder [25], and C HAT GPT [14], is trained on massive amounts of data using general-purpose tasks such as predicting a masked token and predicting the following sentence.”
Token-Mol 1.0: tokenized drug design with large language models
Wang, J., Qin, R., Wang, M., Fang, M., Zhang, Y., Zhu, Y., Su, Q., Gou, Q., Shen, C., Zhang, O., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59628-y · Cited by 29 · see citing works
Text2VRScene: Exploring the Framework of Automated Text-driven Generation System for VR Experience
Yin, Z., Wang, Y., Papatheodorou, T., & Hui, P. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00090 · Cited by 25 · see citing works
“The breakthroughs in Large Language Models (LLMs), like Ope-nAI’s GPT-4, have achieved the advanced capability to comprehend natural language [3,19,27], generate a reasonable, and rational response or even decent coding [8,9,15,17,32].”
MethodAutomated Test Case Repair Using Language Models
Yaraghi, A. S., Holden, D., Kahani, N., & Briand, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3541166 · Cited by 25 · see citing works
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 23 · see citing works
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · see citing works
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 25 · see citing works
“Particularly, abstract model-wise metrics assess the intrinsic properties of the model regardless of subject trustworthiness perspectives, such as the stability of the model and the degree of well-fitting to the distribution of training data.”
BackgroundNon-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22 · see citing works
ChatGPT Code Detection: Techniques for Uncovering the Source of Code
Oedingen, M., Engelhardt, R. C., Denz, R., Hammer, M., & Konen, W. (2024) · AI · doi:10.3390/ai5030053 · Cited by 24 · see citing works
“…an adaptation of DetectGPT for code detection that also operates as a zero-shot detector, by introducing three modifications: ( 1 ) Replacing T5 with the code-specific Incoder-6B [20] for code perturbations, addressing the need for maintaining code’s syntactic and semantic integrity.”
MethodEnhancing large language models for text-to-testcase generation
Alagarsamy, S., Tantithamthavorn, C. K., Takerngsaksiri, W., Arora, C., & Aleti, A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112531 · Cited by 23 · see citing works
Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 40 · see citing works
“3B [2], GPT-3.”
Multi-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 25 · see citing works
Natural Language to Code Generation in Interactive Data Science Notebooks
Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.9 · Cited by 22 · see citing works
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 21 · see citing works
“• InCoder [32] Facebook’s InCoder is a generative model for program synthesis and editing, using the same architecture as the dense models described in [11].”
BackgroundMethodL2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 31 · see citing works
“…on large language models (LLMs) (OpenAI, 2023; Chowdhery et al., 2023; Touvron et al., 2023a), especially those specifically trained for coding (Fried et al., 2023; Nijkamp et al., 2022; Chen et al., 2021; Li et al., 2023), has shown that LLMs trained on a mixture of text and code are able to…”
BackgroundKnowledge-Aware Code Generation with Large Language Models
Huang, T., Sun, Z., Jin, Z. G., Li, G., & Lyu, C. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644418 · Cited by 20 · see citing works
“The emergence of CodeBERT [21] has marked a new epoch in the use of pre-trained LLMs. Pre-trained LLMs such as CodeGen [6], InCoder [22], CodeX [7], AlphaCode [23], Code Llama [24] and StarCoder [25] have yielded significant enhancements in code generation performance.”
MethodA survey on robustness attacks for deep code models
Qu, Y., Huang, S., & Yao, Y. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00464-7 · Cited by 21 · see citing works
“StarCoder’s repeated evaluations revealed that other state-of-the-art LLM4Codes (Fried et al. 2023; Nijkamp et al. 2023) exhibited similarly concerning security levels as Copilot.”
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 22 · see citing works
“With billions of parameters trained on extensive corpora (both general and software engineering specific), LLMs have demonstrated extraordinary competencies in various software engineering tasks, such as code generation [2]– [4], program repair [5]–[7], and documentation generation [8].”
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“However, we lack many models capable of generating code (Black et al., 2021; Fried et al., 2022; Black et al., 2022; Wang & Komatsuzaki, 2021; Biderman et al., 2023b).”
MethodREEF: A Framework for Collecting Real-World Vulnerabilities and Fixes
Wang, C., Li, Z., Pena, Y., Gao, S., Chen, S., Wang, S., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00199 · Cited by 17 · see citing works
“One notable model is Incoder [51], which employs a causal masking training objective to excel in code infilling and synthesis.”
BackgroundSelf-citationIsolating Compiler Bugs by Generating Effective Witness Programs With Large Language Models
Tu, H., Zhou, Z., Jiang, H., Yusuf, I. N. B., Li, Y., & Jiang, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3397822 · Cited by 21 · see citing works
“Infilling models (e.g., CodeGen [32], Incoder [33], and Poly-Coder [34]) are used to insert the most natural code based on bi-directional context (e.g., in the middle of a code snippet), while general models (e.g., LLaMA [35], Alpaca [7], ChatGPT [5], Vicuna [8], and GPT4ALL [9]) target to generate…”
MethodDeep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 17 · see citing works
Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers
Shi, Y., Zhang, H., Wan, C., & Gu, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00005 · Cited by 23 · see citing works
“We selected Incoder [12], Phi-1 [15], StarCoder [25], Wizard-Coder [30], CodeGen2 [34] and CodeLlama [43] from Huggingface 3 .”
- Authors
- Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M.
- Affiliations
- Meta (United States) (US)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.05999
- OpenAlex
- W4224060952
- Cited by
- 187
- References
- 81
- Citations by year
- 2026: 4 · 2025: 52 · 2024: 68 · 2023: 59 · 2022: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program synthesis, bidirectional context, code editing, type inference, comment generation, variable renaming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
