The DOI matches its DataCite record; 72 of the 81 references it cites were found in a registry or an index, and 1 are not checked yet.
InCoder: A Generative Model for Code Infilling and Synthesis
Open accessAbstract
Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, we lack many models capable of generating code (Black et al., 2021;Fried et al., 2022;Black et al., 2022;Wang & Komatsuzaki, 2021;Biderman et al., 2023b).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Fried et al., 2022
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Following existing works evaluating LLM code generation [12, 23] and common evaluation practices over LiveCodeBench and DevEval, we use Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.”
From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Longer contexts enable us to move forward to”
From Context Composing for Full Line Code Completion (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks.”
From Learning Performance-Improving Code Edits (2023) · cites this paper as Fried et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundMethodPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 2×
“For our learning rate scheduler, we use the built-in polynomial decay learning rate scheduler available in [49] with 1500 warmup updates.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Neural models for code generation have either utilized a left-to-right (causal) autoregressive language modeling objective [16, 18] or, as BERT does, a masked language modeling objective [22, 24].”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps [39, 32].”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Self-citationSpace/time trade-offs in hash coding with allowable errors
Bloom, B. (1970) · CACM · doi:10.1145/362686.362692 · Cited by 8,424 · cited 1×
“(6)We implement match checking using a Bloom filter [15] whose keys are: the file extension, number of tokens in the file, and an MD5 hash [53] of the sequence of tokens, which is highly accurate at identifying files with exactly matching token sequences.”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent [76, 65, 48].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5×
“[21] performed best, we found that our model did not benefit from additional examples in the prompt, which we attribute to much smaller size of our model (6.”
BackgroundMethodInfluentialfairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475 · cited 4×
“We use the same architecture as the dense 6.7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
MethodInfluentialWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · ESEC/FSE '11 · doi:10.1145/2025113.2025179 · Cited by 1,445 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 10×
“Neural models for code generation have either utilized a left-to-right (causal) autoregressive language modeling objective [16, 18] or, as BERT does, a masked language modeling objective [22, 24].”
BackgroundMethodInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 1×
“For all three inference methods, we obtain generations from the model using Fairseq’s implementation [47] of top-p (nucleus) sampling [33] with p = 0.”
MethodFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1×
“Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent [76, 65, 48].”
BackgroundCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 2×
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
MethodAutomating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/1926385.1926423 · Cited by 1,039 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundUnified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,000 · cited 2×
“, [18, 45, 4]), we evaluate our model zero-shot, with no explicit documentation string supervision.”
MethodNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1×
“To increase the amount of context that our code model can condition on, the length of documents that the model can generate, and the efficiency of training and inference, we train a byte-level BPE tokenizer [56, 51].”
A systematic evaluation of large language models of codePreprint
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · MAPS@PLDI · doi:10.48550/arxiv.2202.13169 · Cited by 925 · cited 3×
“Large language models trained on vast repositories of code have demonstrated remarkable progress in neural program synthesis and related tasks [18, 10, 68, 46, 21].”
BackgroundMethodCombinatorial sketching for finite programs
Solar-Lezama, A., Tancau, L., Bodík, R., Seshia, S., & Saraswat, V. (2006) · ASPLOS XII · doi:10.1145/1168857.1168907 · Cited by 922 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1×
“0 — Unreleased LaMDA [10, 60, 21] 137 None None ??? — 14.”
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · cited 1×
“We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
MethodGulwani, S., Polozov, O., & Singh, R. (2017) · Software Systems Safety · doi:10.3233/978-1-61499-385-8-43 · Cited by 748 · cited 1×
“Machine Learning for Program Synthesis Program synthesis approaches directly generate programs from a specification of functionality [29].”
MethodDeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundDefending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · cited 1×
“We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
MethodCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 11×
“We also include the performance of the supervised baseline as reported in the CodeXGLUE paper.”
MethodInfluentialThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 2×
“The Pile [26], which was used to train recent generative code models that we compare to in Section 6, also contains these questions and answers but does not contain comments.”
MethodCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 4×
“CodeXGLUE cloze is created from CodeSearchNet to evaluate CodeBERT and consists of a short natural language description followed by code in several programming languages.”
MethodInfluentialMask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464 · cited 1×
“Finally, our model lays a foundation for future work on supervised infilling & editing via model fine-tuning, as well as performing iterative decoding, where the model can be used to refine its own output [27].”
MethodSelf-citationSEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 448 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodDeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 8×
“We compare model pass@1 scores on the HumanEval [18] and MBPP [10] left-to-right synthesis benchmarks, with results in Table 7.”
BackgroundMethodInfluentialCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 5×
“While differences in details of the Transformer model architectures, datasets, and training procedures across papers and experimental setups make a rigorous comparison impossible, we note that our model achieves roughly comparable performance on the HumanEval metrics to CodeGen-Multi [46], which is also a ∼6B paramete…”
BackgroundMethodInfluentialInsertion Transformer: Flexible Sequence Generation via Insertion OperationsPreprint
Stern, M., Chan, W., Kiros, J., & Uszkoreit, J. (2019) · International Conference on Machine Learning · doi:10.48550/arxiv.1902.03249 · Cited by 300 · cited 1×
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundThe adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · cited 1×
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
BackgroundCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · cited 2×
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
MethodEfficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.10684 · Cited by 223 · cited 1×
“7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
BackgroundSelf-citationEnabling Language Models to Fill in the Blanks
Donahue, C., Lee, M., & Liang, P. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2005.05339 · Cited by 222 · cited 1×
“[23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al.”
BackgroundLearning to mine aligned code and natural language pairs from stack overflow
Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196408 · Cited by 218 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundHellendoorn, V. J., Bird, C., Barr, E. T., & Allamanis, M. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236051 · Cited by 216 · cited 2×
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodPyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.03150 · Cited by 168
Allamanis, M., Barr, E. T., Ducousso, S., & Gao, Z. (2020) · ACM-SIGPLAN Symposium on Programming Language Design and Implementation · doi:10.1145/3385412.3385997 · Cited by 156 · cited 1×
“Some recent work has developed supervised machine learning approaches for predicting type annotations for dynamically-typed languages including Python [69, 6, 50] and TypeScript [31, 66, 36].”
MethodAdapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 152
Self-citationScaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · cited 1×
“Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps [39, 32].”
BackgroundPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 145
TypeWriter: neural type prediction with search-based validation
Pradel, M., Gousios, G., Liu, J., & Chandra, S. (2019) · ESEC/SIGSOFT FSE · doi:10.1145/3368089.3409715 · Cited by 141 · cited 9×
“We compare our zero-shot model to one such approach for Python, TypeWriter [50], which combines a neural architecture for type hint prediction with a search-based incremental type validation procedure.”
BackgroundMethodInfluentialLambdaNet: Probabilistic Type Inference using Graph Neural NetworksPreprint
Wei, J., Goyal, M., Durrett, G., & Dillig, I. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2005.02161 · Cited by 120 · cited 3×
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodDOBF: A Deobfuscation Pre-Training Objective for Programming LanguagesPreprint
Rozière, B., Lachaux, M.-A., Szafraniec, M., & Lample, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07492 · Cited by 113 · cited 2×
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundPython probabilistic type inference with natural language support
Xu, Z.-G., Zhang, X., Chen, L., Pei, K.-X., & Xu, B.-W. (2016) · SIGSOFT FSE · doi:10.1145/2950290.2950343 · Cited by 108 · cited 1×
“Some recent work has developed supervised machine learning approaches for predicting type annotations for dynamically-typed languages including Python [69, 6, 50] and TypeScript [31, 66, 36].”
MethodSPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundCodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86 · cited 1×
“, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
CLN2INV: Learning Loop Invariants with Continuous Logic NetworksPreprint
Ryan, G., Wong, J., Yao, J., Gu, R.-H., & Jana, S. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1909.11542 · Cited by 82 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodGraph-based, Self-Supervised Program Repair from Diagnostic Feedback
Yasunaga, M., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10636 · Cited by 81 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodRestoring ancient text using deep learning: a case study on Greek epigraphy
Assael, Y., Sommerschield, T., & PRAG, J. R. W. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1668 · Cited by 78 · cited 1×
“, editing sentences [57], restoring ancient text [9], and fixing bugs in source code.”
BackgroundAutoPandas: neural-backed generators for program synthesis
Bavishi, R., Lemieux, C., Fox, R., Sen, K., & Stoica, I. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360594 · Cited by 73 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundDeduplicating Training Data Mitigates Privacy Risks in Language Models
Kandpal, N., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2202.06539 · Cited by 68 · cited 1×
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
BackgroundSelf-citationGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67
KERMIT: Generative Insertion-Based Modeling for SequencesPreprint
Chan, W., Kitaev, N., Guu, K., Stern, M., & Uszkoreit, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01604 · Cited by 64 · cited 1×
“Unfortunately, standard left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans [17, 22, 52, 54].”
BackgroundJuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
Agashe, R., Iyer, S. V., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1546 · Cited by 56 · cited 2×
“We additionally removed repositories in the evaluation sets of the JuICe tasks [1], although we did not evaluate our model on these tasks in this present work.”
BackgroundMethodSelf-citationBreak-It-Fix-It: Unsupervised Learning for Program RepairPreprint
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 48 · cited 1×
“This includes methods to infer variable types [50, 66], generate unit tests [25], repair programs [30, 70, 20, 71], and verify program correctness [55].”
MethodEfficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47 · cited 1×
“In addition, concurrent to our work, OpenAI developed a fill-in-the-middle (FIM) training objective similar to the causal masking objective we use, trained code models with it, and evaluated on the HumanEval infilling tasks we introduce here (Bavarian et al., 2022).”
MethodLearning type annotation: is big data enough?
Jesse, K., Dévanbu, P., & Ahmed, T. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473135 · Cited by 46 · cited 2×
“Our filtering is similar to past work on generative models of code [18, 46, 68]: we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores, or appear to be automatically generated, which we d…”
MethodCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 6×
“Most related to our approach is the work of Donahue et al. [23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al. [7], who train an infilling-capable, AST-structured generative model of code on a smaller scale.”
BackgroundMethodInfluentialSelf-citationHTLM: Hyper-Text Pre-Training and Prompting of Language Models
Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2107.06955 · Cited by 35 · cited 1×
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundSelf-citationSpreadsheetCoder: Formula Prediction from Semi-structured ContextPreprint
Chen, X., Maniatis, P., Singh, R., Sutton, C. A., Dai, H., Lin, M., & Zhou, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15339 · Cited by 21 · cited 1×
“, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
BackgroundXL-Editor: Post-editing Sentences with XLNetPreprint
Shih, Y.-S., Chang, W.-C., & Yang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10479 · Cited by 13 · cited 1×
“, editing sentences [57], restoring ancient text [9], and fixing bugs in source code.”
BackgroundReflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models
West, P., Lu, X., Holtzman, A., Bhagavatula, C., Hwang, J. D., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.114 · Cited by 12 · cited 1×
“Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
BackgroundDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2107.06499 · Cited by 11 · cited 1×
“Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
Background
Research works printed but not linked: 4
- Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In ICML , pp. 245-256. PMLR, 2020.
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In ICML , 2020.
- Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Transformers . O'Reilly Media, Inc., 2022.
- John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In AAAI , 1996.
Cited by
Sorted by citations
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
“For example, all experiments on admissible sets improve on the previous best capacity lower bound, with 60% of experiments on (12,7) I finding a full-size admissible set.”
BackgroundCode Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“Our code-training for 7B, 13B and 70B Code Llama models features a multitask objective (Fried et al., 2023) consisting of both autoregressive and causal infilling prediction, enabling applications such as real-time completion in source code editors or docstring generation.”
BackgroundMethodInfluentialLarge language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“For example, Murali et al. [37] reported the experience of deploying CodeCompose, a code completion tool based on the Incoder LLM [38], at Meta.”
MethodAutomated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
“• INCODER [33] A model designed for code infilling by adopting a causal masking objective [45].”
Influential
Metadata
- Authors
- Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M.
- Affiliations
- Meta (United States) (US)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.05999
- OpenAlex
- W4224060952
- Cited by
- 187
- References
- 81
- Citations by year
- 2026: 4 · 2025: 52 · 2024: 68 · 2023: 59 · 2022: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program synthesis, bidirectional context, code editing, type inference, comment generation, variable renaming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
