On the naturalness of software
The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations - and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The IntelliCode Compose system uses the term click through rate (CTR) for this and reports a value of 10% in online trials.20”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [20]
In the citing paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [11]
Citations
Cites
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Statistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 871
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Mining version histories to guide software changes
Zimmermann, T., Weibgerber, P., Diehl, S., & Zeller, A. (2004) · Proceedings. 26th International Conference on Software Engineering · doi:10.1109/icse.2004.1317478 · Cited by 322
Statistical language model adaptation: review and perspectives
Bellegarda, J. R. (2004) · Speech Communication · doi:10.1016/j.specom.2003.08.002 · Cited by 297
Cited by
Items 101–200 of 834
An Empirical Study on the Usage of BERT Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00024 · Cited by 78
Learning Deep Semantics for Test Completion
Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., & Gligoric, M. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00178 · Cited by 57
Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
Bhatia, S., & Singh, R. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06129 · Cited by 63
A deep language model for software codePreprint
Dam, H. K., Tran, T., & Pham, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02715 · Cited by 125
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124
“Code completion in IDEs using language models was first proposed in Hindle et al., 9 and today neural synthesis tools such as GitHub Copilot, CodeWhisperer, and TabNine suggest code snippets within an IDE with the explicitly stated intention to increase a user’s productivity. a Alternatively, we…”
MethodQlose: Program Repair with Quantitative Objectives
D’Antoni, L., Samanta, R., & Singh, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41540-6_21 · Cited by 123
A general path-based representation for predicting program properties
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2018) · ACM SIGPLAN Notices · doi:10.1145/3296979.3192412 · Cited by 121
Döllner, J. (2020) · PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science · doi:10.1007/s41064-020-00102-3 · Cited by 84
Flow2Vec: value-flow-based precise code embedding
Sui, Y., Cheng, X. Y., Zhang, G., & Wang, H. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428301 · Cited by 120
Neural Sketch Learning for Conditional Program Generation
Murali, V., Qi, L., Chaudhuri, S., & Jermaine, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05698 · Cited by 47
Type4Py: practical deep similarity learning-based type inference for python
Mir, A. M., Latoškinas, E., Proksch, S., & Gousios, G. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510124 · Cited by 117
Automatic recovery from runtime failures
Carzaniga, A., Gorla, A., Mattavelli, A., Perino, N., & Pezzè, M. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606624 · Cited by 116
Pandl, K. D., Thiebes, S., Schmidt-Kraepelin, M., & Sunyaev, A. (2020) · IEEE Access · doi:10.1109/access.2020.2981447 · Cited by 115
Parameter-free probabilistic API mining across GitHub
Fowkes, J., & Sutton, C. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950319 · Cited by 99
Martinez, M., Weimer, W., & Monperrus, M. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591114 · Cited by 76
Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs
Wang, Y., & Li, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17650 · Cited by 108
Just-In-Time Defect Identification and Localization: A Two-Phase Framework
Yan, M. L., Xia, X., Fan, Y., Hassan, A. E., Lo, D., & Li, S. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2978819 · Cited by 106
NatGen: generative pre-training by “naturalizing” source code
Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P. T., & Ray, B. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549162 · Cited by 105
“Statistical models of the łnaturalness" of code [33] have proven useful for a range of Software Engineering tasks [7, 50], including code generation [10], repair [15, 61], summarization [40], retrieval [47], and clone detection [20, 66].”
BackgroundSoftware defect prediction via LSTM
Deng, J., Lu, L., & Qiu, S. (2020) · IET Software · doi:10.1049/iet-sen.2019.0149 · Cited by 104
Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost
Afshan, S., McMinn, P., & Stevenson, M. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.11 · Cited by 103
Natural Language Models for Predicting Programming Comments
Movshovitz‐Attias, D., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6475913 · Cited by 100
PanGu-Coder: Program Synthesis with Function-Level Language ModelingPreprint
Christopoulou, F., Λάμπουρας, Γ., Gritta, M., Zhang, G., Guo, Y., Li, Z., Zhang, Q., Xiao, M., Shen, B., Li, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.11280 · Cited by 102
Stack Overflow in Github: Any Snippets There?
Yang, D., Martins, P., Saini, V., & Lopes, C. V. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.13 · Cited by 101
A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning
Liu, F., Li, G., Wei, B., Xia, X., Fu, Z., & Jin, Z. (2020) · Proceedings of the 28th International Conference on Program Comprehension · doi:10.1145/3387904.3389261 · Cited by 84
Code Search is All You Need? Improving Code Suggestions with Code Search
Chen, J., Hu, X., Li, Z., Gao, C., Xia, X., & Lo, D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639085 · Cited by 101
Learning Python Code Suggestion with a Sparse Pointer Network
Bhoopchand, A., Rocktäschel, T., Barr, E. T., & Riedel, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08307 · Cited by 60
Self-citationA Literature Study of Embeddings on Source CodePreprint
Chen, Z., & Monperrus, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03061 · Cited by 99
Learning Syntactic Program Transformations from Examples
Rolim, R., Soares, G., D’Antoni, L., Polozov, O., Gulwani, S., Gheyi, R., Suzuki, R., & Hartmann, B. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.44 · Cited by 98
Fast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95
Mining Android App Usages for Generating Actionable GUI-Based Execution Scenarios
Linares‐Vásquez, M., White, M., Bernal-Cárdenas, C., Moran, K., & Poshyvanyk, D. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.18 · Cited by 95
Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion
Cheng, W., Wu, Y., & Hu, W. P. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.431 · Cited by 92
Generating Commit Messages from Diffs using Pointer-Generator Network
Liu, Q., Liu, Z., Zhu, H., Fan, H., Du, B., & Qian, Y. (2019) · 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00056 · Cited by 64
Code vectors: understanding programs through embedded abstracted symbolic traces
Henkel, J., Lahiri, S. K., Liblit, B., & Reps, T. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236085 · Cited by 90
Assessing the Generalizability of Code2vec Token Embeddings
Kang, H. J., Bissyandé, T. F., & Lo, D. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00011 · Cited by 90
Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 40
Suggesting natural method names to check name consistencies
Nguyen, S., Phan, H., Le, T., & Nguyen, T. N. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380926 · Cited by 89
CodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86
Patching as translation: the data and the metaphor
Ding, Y., Ray, B., Dévanbu, P., & Hellendoorn, V. J. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416587 · Cited by 84
Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?
Peng, Y., Li, S., Gu, W., Li, Y., Wang, W., Gao, C., & Lyu, M. R.-T. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3197063 · Cited by 83
Chen, C., Xing, Z., Liu, Y., & Xiong, K. O. L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2896123 · Cited by 82
Montage: A Neural Network Language Model-Guided JavaScript Engine FuzzerPreprint
Lee, S., Han, H.-S., Cha, S. K., & Son, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04107 · Cited by 80
TRACED: Execution-aware Pre-training for Source Code
Ding, Y., Steenhoek, B. J., Pei, K., Kaiser, G. E., Le, W., & Ray, B. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608140 · Cited by 79
When Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76
“…without exception, code completion tools are evaluated against synthetic benchmarks , typically produced by adding “holes” to existing code (e.g., removing an identifier) and tasking the model with predicting the correct expression for that location using the remaining context [4], [6], [9]–[12].”
BackgroundMethodInfluentialWong, M. F., & Tan, C. W. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3524104 · Cited by 76
Challenging Machine Learning-Based Clone Detectors via Semantic-Preserving Code Transformations
Zhang, W., Guo, S., Zhang, H., Sui, Y., Xue, Y., & Xu, Y. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3240118 · Cited by 74
Understanding neural code intelligence through program simplification
Rabin, M. R. I., Hellendoorn, V. J., & Alipour, M. A. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468539 · Cited by 73
Machine Learning Based Recommendation of Method Names: How Far are We
Jiang, L. C., Liu, H., & Jiang, H. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00062 · Cited by 72
SOSRepair: Expressive Semantic Search for Real-World Program Repair
Afzal, A., Motwani, M., Stolee, K. T., Brun, Y., & Goues, C. L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2944914 · Cited by 71
TacTok: semantics-aware proof synthesis
First, E., Brun, Y., & Guha, A. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428299 · Cited by 41
Improving Bug Localization with an Enhanced Convolutional Neural Network
Xiao, Y., Keung, J., Mi, Q., & Bennin, K. E. (2017) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.40 · Cited by 71
Kondo, M., Oliva, G. A., Jiang, Z. M., Hassan, A. E., & Mizuno, O. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09852-5 · Cited by 71
Ying, A. T. T., & Robillard, M. P. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494587 · Cited by 69
Current Trends in Source Code Analysis, Plagiarism Detection and Issues of Analysis Big Datasets
Ďuračík, M., Kršák, E., & Hrkút, P. (2017) · Procedia Engineering · doi:10.1016/j.proeng.2017.06.024 · Cited by 67
Deep Learning Based Program Generation From Requirements Text: Are We There Yet?
Liu, H., Shen, M., Zhu, J., Niu, N., Li, G., & Zhang, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3018481 · Cited by 66
Ernst, M. D. (2017) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.snapl.2017.4 · Cited by 40
Learning programs from noisy data
Raychev, V., Bielik, P., Vechev, M., & Krause, A. (2016) · ACM SIGPLAN Notices · doi:10.1145/2914770.2837671 · Cited by 65
SWordNet: Inferring semantically related words from software context
Yang, J., & Tan, L. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9264-x · Cited by 64
Will They Like This? Evaluating Code Contributions with Language Models
Hellendoorn, V. J., Dévanbu, P., & Bacchelli, A. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.22 · Cited by 64
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05734 · Cited by 39
Autofolding for Source Code Summarization
Fowkes, J., Chanthirasegaran, P., Ranca, R., Allamanis, M., Lapata, M., & Sutton, C. A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2664836 · Cited by 63
Learning Quick Fixes from Code Repositories
Rolim, R., Soares, G., Gheyi, R., Barik, T., & D’Antoni, L. (2018) · Anais do Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.48550/arxiv.1803.03806 · Cited by 63
Public git archive: a big code dataset for all
Markovtsev, V., & Long, W. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196464 · Cited by 39
Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts
Bavishi, R., Pradel, M., & Sen, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05193 · Cited by 42
Learning to Generate Corrective Patches using Neural Machine TranslationPreprint
Hata, H., Shihab, E., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07170 · Cited by 63
From Copilot to Pilot: Towards AI Supported Software DevelopmentPreprint
Pudari, R., & Ernst, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.04142 · Cited by 63
AI-Driven Development Is Here: Should You Worry?
Ernst, N. A., & Bavota, G. (2022) · IEEE Software · doi:10.1109/ms.2021.3133805 · Cited by 62
Commit2Vec: Learning Distributed Representations of Code Changes
Lozoya, R. C., Baumann, A., Sabetta, A., & Michele Bezzi, A. (2021) · SN Computer Science · doi:10.1007/s42979-021-00566-z · Cited by 61
Tian, H., Liu, K., Li, Y., Kaboré, A. K., Koyuncu, A., Habib, A., Li, L., Wen, J., Klein, J., & Bissyandé, T. F. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576039 · Cited by 60
Detecting Duplicate Bug Reports with Convolutional Neural Networks
Xie, Q., Wen, Z., Zhu, J., Gao, C., & Zheng, Z. (2018) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2018.00056 · Cited by 60
On the Effectiveness of Transfer Learning for Code Search
Salza, P., Schwizer, C., Gu, J., & Gall, H. C. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3192755 · Cited by 43
Cross-language code search using static and dynamic analyses
Mathew, G., & Stolee, K. T. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468538 · Cited by 59
Modular Tree Network for Source Code Representation Learning
Wang, W., Li, G., Shen, S., Xia, X., & Jin, Z. G. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3409331 · Cited by 58
Unsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57
Exploring the use of deep learning for feature location
Corley, C. S., Damevski, K., & Kraft, N. A. (2015) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2015.7332513 · Cited by 57
Diversity-driven automated formal verification
First, E., & Brun, Y. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510138 · Cited by 33
Deep API LearningPreprint
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08535 · Cited by 56
CSCC: Simple, Efficient, Context Sensitive Code Completion
Asaduzzaman, M., Roy, C. K., Schneider, K. A., & Hou, D. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.29 · Cited by 55
An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering
Li, Y., Shi, J., & Zhang, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3385682 · Cited by 55
Enabling Clone Detection For Ethereum Via Smart Contract Birthmarks
Liu, H., Yang, Z., Jiang, Y., Zhao, W., & Sun, J. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00024 · Cited by 54
Detecting argument selection defects
Rice, A. C., Aftandilian, E. E., Jaspan, C. N. C., Johnston, E., Pradel, M., & Arroyo-Paredes, Y. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133928 · Cited by 53
The Essence of Software Engineering
Gruhn, V., & Striemer, R. (2018) · Springer eBooks · doi:10.1007/978-3-319-73897-0 · Cited by 53
Holistic Combination of Structural and Textual Code Information for Context Based API Recommendation
Chen, C., Peng, X., Xing, Z., Sun, J., Wang, X., Zhao, Y., & Zhao, W. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3074309 · Cited by 52
SSPCatcher: Learning to catch security patches
Sawadogo, A. D., Bissyandé, T. F., Moha, N., Allix, K., Klein, J., Li, L., & Traon, Y. L. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10168-9 · Cited by 52
Code Execution with Pre-trained Language Models
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.308 · Cited by 51
Artificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 51
How Practitioners Expect Code Completion?
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616280 · Cited by 50
Tian, H., Tang, X., Habib, A., Wang, S., Liu, K., Xia, X., Klein, J., & Bissyandé, T. F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556914 · Cited by 50
Building Language Models for Text with Named Entities
Parvez, M. R., Chakraborty, S., Ray, B., & Chang, K.-W. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1221 · Cited by 36
Towards a Natural Perspective of Smart Homes for Practical Security and Safety Analyses
Manandhar, S., Moran, K., Kafle, K., Tang, R., Poshyvanyk, D., & Nadkarni, A. (2020) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40000.2020.00062 · Cited by 50
Does BLEU Score Work for Code Migration?
Tran, N., Tran, H., Nguyen, S., Nguyen, H., & Nguyen, T. (2019) · 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00034 · Cited by 49
A theory of dual channel constraints
Casalnuovo, C., Barr, E. T., Dash, S. K., Dévanbu, P., & Morgan, E. (2020) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3377816.3381720 · Cited by 49
Self-citationSeCNN: A semantic CNN parser for code comment generation
Li, Z., Wu, Y., Peng, B., Chen, X., Sun, Z., Liu, Y., & Yu, D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111036 · Cited by 49
Boosting Neural Networks to Decompile Optimized Binaries
Cao, Y., Liang, R., Chen, K., & Hu, P. (2022) · Annual Computer Security Applications Conference · doi:10.1145/3564625.3567998 · Cited by 48
A deep neural network language model with contexts for source code
Nguyen, A. T., Nguyen, T. D., Phan, H., & Nguyen, T. N. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330220 · Cited by 48
Memorization and Generalization in Neural Code Intelligence Models
Rabin, M. R. I., Hussain, A., Alipour, M. A., & Hellendoorn, V. J. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107066 · Cited by 48
Studying LLM Performance on Closed- and Open-source DataPreprint
Ahmed, T., Bird, C., Dévanbu, P., & Chakraborty, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.15100 · Cited by 48
What is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47
“Some support for this hypothesis comes from research that used n -gram models to build a code completion engine for Java that outperformed Eclipse’s completion feature (Hindle et al., 2012, 2016).”
MethodBilateral Dependency Neural Networks for Cross-Language Algorithm Classification
Bui, N. D. Q., Yu, Y., & Jiang, L. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667995 · Cited by 47
SeTransformer: A Transformer-Based Code Semantic Parser for Code Comment Generation
Li, Z., Wu, Y., Peng, B., Chen, X., Sun, Z., Liu, Y., & Doyle, P. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3154773 · Cited by 47
The growing cost of deep learning for source code
Hellendoorn, V. J., & Sawant, A. A. (2021) · Communications of the ACM · doi:10.1145/3501261 · Cited by 46
Metadata
- Authors
- Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- The University of Texas at Dallas (US)
- Venue
- 2012 34th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 837-847
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W3146720657
- Cited by
- 425
- References
- 39
- Citations by year
- 2026: 19 · 2025: 24 · 2024: 36 · 2023: 44 · 2022: 49 · 2021: 50 · 2020: 48 · 2019: 46 · 2018: 22 · 2017: 35 · 2016: 13 · 2015: 23 · 2014: 2 · 2013: 14
- FWCI
- 9.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support, natural language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.