On the naturalness of software
The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations - and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The IntelliCode Compose system uses the term click through rate (CTR) for this and reports a value of 10% in online trials.20”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [20]
In the citing paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [11]
Citations
Cites
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Statistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 871
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Mining version histories to guide software changes
Zimmermann, T., Weibgerber, P., Diehl, S., & Zeller, A. (2004) · Proceedings. 26th International Conference on Software Engineering · doi:10.1109/icse.2004.1317478 · Cited by 322
Statistical language model adaptation: review and perspectives
Bellegarda, J. R. (2004) · Speech Communication · doi:10.1016/j.specom.2003.08.002 · Cited by 297
Cited by
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
code2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962
Convolutional Neural Networks over Tree Structures for Programming Language Processing
Mou, L., Li, G., Zhang, L., Wang, T., & Jin, Z. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10139 · Cited by 675
A Novel Neural Source Code Representation Based on Abstract Syntax Tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., & Liu, X. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00086 · Cited by 655
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“From the pioneering work of Hindle et al. on the naturalness of software [43], we know that programmers write code (and languages enforce code writing styles), that make code highly predictable.”
BackgroundCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950334 · Cited by 485
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
“ation to detect programming language. Given N lang one-dimensional monolingual arrays of subtokens, the training dataset for the double-headed GPT-C is extracted according to an algorithm detailed in [7]. As seen, language specific control codes provide necessary supervision to constrain language generation for each sp…”
MethodSoftware Defect Prediction via Convolutional Neural Network
Li, J., He, P., Zhu, J., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2017.42 · Cited by 462
Software Vulnerability Detection Using Deep Neural Networks: A Survey
Lin, G., Wen, S., Han, Q.-L., Zhang, J., & Xiang, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2993293 · Cited by 457
SourcererCC: scaling code clone detection to big-code
Sajnani, H., Saini, V., Svajlenko, J., Roy, C. K., & Lopes, C. V. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884877 · Cited by 447
Learning and Evaluating Contextual Embedding of Source CodePreprint
Kanade, A., Maniatis, P., Balakrishnan, G., & Shi, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00059 · Cited by 438
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
DeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Learning natural coding conventions
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635883 · Cited by 376
Self-citationAn Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370
Suggesting accurate method and class names
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786849 · Cited by 369
Self-citationDeepBugs: a learning approach to name-based bug detection
Pradel, M., & Sen, K. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3276517 · Cited by 342
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313
Mining source code repositories at massive scale using language modeling
Allamanis, M., & Sutton, C. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624029 · Cited by 313
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00181 · Cited by 296
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284
Deep code comment generation with hybrid lexical and syntactical information
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09730-9 · Cited by 278
Code Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., & King, I. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/578 · Cited by 273
Summarizing Source Code with Transferred API Knowledge
Hu, X., Li, G., Xia, X., Lo, D., Lu, S., & Jin, Z. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/314 · Cited by 263
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
Automatically generating commit messages from diffs using neural machine translation
Jiang, S., Armaly, A., & McMillan, C. (2017) · 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115626 · Cited by 255
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884848 · Cited by 252
Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation
Oda, Y., Fudaba, H., Neubig, G., Hata, H., Sakti, S., Toda, T., & Nakamura, S. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.36 · Cited by 250
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240
Personalized defect prediction
Jiang, T., Tan, L., & Kim, S. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693087 · Cited by 240
Learning Memory Access PatternsPreprint
Hashemi, M., Swersky, K., Smith, J., Ayers, G., Litz, H., Chang, J., Kozyrakis, C., & Ranganathan, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02329 · Cited by 240
A perspective on the evolution of live programming
Tanimoto, S. L. (2013) · doi:10.1109/live.2013.6617346 · Cited by 232
DeepJIT: An End-to-End Deep Learning Framework for Just-in-Time Defect Prediction
Hoang, T., Khanh Dam, H., Kamei, Y., Lo, D., & Ubayashi, N. (2019) · 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00016 · Cited by 223
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 223
Panichella, A., Dit, B., Oliveto, R., Di Penta, M., Poshynanyk, D., & De Lucia, A. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606598 · Cited by 216
Learning to Prove Theorems via Interacting with Proof AssistantsPreprint
Yang, K., & Deng, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09381 · Cited by 208
Learning syntactic program transformations from examples.
de Sousa, R. R. (2018) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 208
A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software
Ponta, S. E., Plate, H., Sabetta, A., Michele Bezzi, A., & Dangremont, C. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00064 · Cited by 207
Panichella, A., Dit, B., Oliveto, R., Penta, M. D., Poshynanyk, D., & Lucia, A. D. (2013) · Archivio Istituzionale della Ricerca - Università degli Studi del Molise · Cited by 206
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Xia, C. S., & Zhang, L. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680323 · Cited by 203
Toward Deep Learning Software Repositories
White, M., Vendome, C., Linares-Vasquez, M., & Poshyvanyk, D. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.38 · Cited by 200
Bug Localization with Combination of Deep Learning and Information Retrieval
Lam, A. N., Nguyen, A. T., Nguyen, H. A., & Nguyen, T. N. (2017) · 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2017.24 · Cited by 194
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for CodePreprint
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 193
Toward deep learning software repositories
White, M., Vendome, C., & Poshyvanyk, D. (2015) · Cited by 192
Graph-Based Statistical Language Model for Code
Nguyen, A. T., & Nguyen, T. N. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.336 · Cited by 190
Elixir: Effective object-oriented program repair
Saha, R. K., Lyu, Y., Yoshida, H., & Prasad, M. R. (2017) · 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115675 · Cited by 189
Lexical statistical machine translation for language migration
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494584 · Cited by 189
Multi-task Learning based Pre-trained Language Model for Code Completion
Liu, F., Li, G., Zhao, Y., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416591 · Cited by 188
A survey on the use of topic models when mining software repositories
Chen, T.-H., Thomas, S. W., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9402-8 · Cited by 187
VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python
Wartschinski, L., Noller, Y., Vogel, T., Kehrer, T., & Grunske, L. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2021.106809 · Cited by 187
Automatic Feature Learning for Predicting Vulnerable Software Components
Dam, H. K., Tran, T., Pham, T., Ng, S. W., Grundy, J., & Ghose, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2881961 · Cited by 184
PyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 184
Code smell detection by deep direct-learning and transfer-learning
Sharma, T., Efstathiou, V., Λουρίδας, Π., & Spinellis, D. D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110936 · Cited by 184
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
Hellendoorn, V. J., Bird, C., Barr, E. T., & Allamanis, M. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236051 · Cited by 181
Self-citationStructured Generative Models of Natural Source CodePreprint
Maddison, C. J., & Tarlow, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0514 · Cited by 180
CODIT: Code Editing With Tree-Based Neural Models
Chakraborty, S., Ding, Y., Allamanis, M., & Ray, B. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3020502 · Cited by 179
Logram: Efficient Log Parsing Using n-Gram Dictionaries
Dai, H., Li, H., Chen, C. S., Shang, W., & Chen, T.-H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3007554 · Cited by 170
Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges
Le, T. H. M., Chen, H., & Babar, M. A. (2021) · ACM Computing Surveys · doi:10.1145/3383458 · Cited by 170
NL2Type: Inferring JavaScript Function Types from Natural Language Information
Malik, R. S., Patra, J., & Pradel, M. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00045 · Cited by 169
DIRE: A Neural Approach to Decompiled Identifier Naming
Lacomis, J., Yin, P., Schwartz, E. J., Allamanis, M., Goues, C. L., Neubig, G., & Vasilescu, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00064 · Cited by 168
Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise
Floyd, B., Santander, T., & Weimer, W. R. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.24 · Cited by 167
API code recommendation using statistical learning from fine-grained changes
Nguyen, A. T., Hilton, M., Codoban, M., Nguyen, H. A., Mast, L., Rademacher, E., Nguyen, T. N., & Dig, D. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950333 · Cited by 163
Mining Fix Patterns for FindBugs Violations
Liu, K., Kim, D., Bissyandé, T. F., Yoo, S., & Traon, Y. L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2884955 · Cited by 163
A study of repetitiveness of code changes in software evolution
Nguyen, H. A., Nguyen, A. T., Nguyen, T. T., Nguyen, T. N., & Rajan, H. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693078 · Cited by 162
PalmTree: Learning an Assembly Language Model for Instruction Embedding
Li, X., Qu, Y., & Yin, H. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484587 · Cited by 161
SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics
Ye, H., Martínez, M., Luo, X., Zhang, T., & Monperrus, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556926 · Cited by 161
Neural Transfer Learning for Repairing Security Vulnerabilities in C Code
Chen, Z., Kommrusch, S., & Monperrus, M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3147265 · Cited by 160
Building Program Vector Representations for Deep Learning
Peng, H. M., Mou, L., Li, G., Liu, Y., Zhang, L., & Jin, Z. G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-25159-2_49 · Cited by 160
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 160
(2014) · doi:10.1145/2635868.2635901 · Cited by 159
Exploring API Embedding for API Usages and Applications
Nguyen, T. D., Nguyen, A. T., Phan, H. D., & Nguyen, T. N. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.47 · Cited by 159
Allamanis, M., Barr, E. T., Ducousso, S., & Gao, Z. (2020) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3385412.3385997 · Cited by 157
Self-citationTian, H., Liu, K., Kaboré, A. K., Koyuncu, A., Li, L., Klein, J., & Bissyandé, T. F. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416532 · Cited by 156
Automated library recommendation
Thung, F., Lo, D., & Lawall, J. (2013) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671293 · Cited by 155
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150
Self-citationLearning to Spot and Refactor Inconsistent Method Names
Liu, K., Kim, D., Bissyandé, T. F., Kim, T.-Y., Kim, K., Koyuncu, A., Kim, S., & Traon, Y. L. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00019 · Cited by 144
Commit Message Generation for Source Code Changes
Xu, S., Yao, Y., Feng, X., Gu, T., Tong, H., & Lü, J. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/552 · Cited by 144
Vulnerability Prediction From Source Code Using Machine Learning
Bilgin, Z., Ersoy, M. A., Soykan, E. Ü., Tomur, E., Çomak, P., & Karaçay, L. (2020) · IEEE Access · doi:10.1109/access.2020.3016774 · Cited by 143
Graph-based statistical language model for code
Nguyen, A. T., & Nguyen, T. N. (2016) · Cited by 142
Learning programs from noisy data
Raychev, V., Bielik, P., Vechev, M., & Krause, A. (2016) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/2837614.2837671 · Cited by 141
An Improved CNN Model for Within-Project Software Defect Prediction
Pan, C., Lu, M., Xu, B., & Gao, H. (2019) · Applied Sciences · doi:10.3390/app9102138 · Cited by 138
CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context
Ding, Y., Wang, Z., Ahmad, W. U., Ramanathan, M. K., Nallapati, R. M., Bhatia, P., Roth, D., & Xiang, B. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/55z6pyuui3ij · Cited by 138
Predicting Defective Lines Using a Model-Agnostic Technique
Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023177 · Cited by 137
An Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 134
DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing
Liu, X., Li, X., Prajapati, R., & Wu, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011044 · Cited by 132
Exploring Software Naturalness through Neural Language ModelsPreprint
Buratti, L., Pujar, S., Bornea, M. A., McCarley, S., Zheng, Y., Rossiello, G., Morari, A., Laredo, J., Thost, V., Zhuang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12641 · Cited by 131
Language-Agnostic Representation Learning of Source Code from Structure and ContextPreprint
Zügner, D., Kirschstein, T., Catasta, M., Leskovec, J., & Günnemann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11318 · Cited by 131
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Mining fine-grained code changes to detect unknown change patterns
Negara, S., Codoban, M., Dig, D., & Johnson, R. E. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568317 · Cited by 129
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 129
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Loyola, P., Marrese-Taylor, E., & Matsuo, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2045 · Cited by 128
A Practical Approach to the Automatic Classification of Security-Relevant Commits
Sabetta, A., & Michele Bezzi, A. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00058 · Cited by 127
Metadata
- Authors
- Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- The University of Texas at Dallas (US)
- Venue
- 2012 34th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 837-847
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W3146720657
- Cited by
- 425
- References
- 39
- Citations by year
- 2026: 19 · 2025: 24 · 2024: 36 · 2023: 44 · 2022: 49 · 2021: 50 · 2020: 48 · 2019: 46 · 2018: 22 · 2017: 35 · 2016: 13 · 2015: 23 · 2014: 2 · 2013: 14
- FWCI
- 9.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support, natural language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.