The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
On the naturalness of software
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural , in the sense that it is created by humans at work, with all the attendant constraints and limitations---and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether (a) code can be usefully modeled by statistical language models and (b) such models can be leveraged to support software engineers. Using the widely adopted n -gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very regular, and, in fact, even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
On The Language Instinct: (412952005-009)
Pinker, S., & American Psychological Association (1994) · PsycEXTRA Dataset · doi:10.1037/e412952005-009 · Cited by 5,646
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Knuth, D. E. (1984) · The Computer Journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,467
Statistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Mining Version Histories to Guide Software Changes
Zimmermann, T., Member, S., Weißgerber, P., Diehl, S., Zeller, A., & Ieee Computer Society (2004) · Cited by 494
Building a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 489
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Research works printed but not linked: 6
- 10.5555/2819009.2819143 · doi:10.5555/2819009.2819143
- 10.5555/1734086 · doi:10.5555/1734086
- 10.1145/1062455.1062526 · doi:10.1145/1062455.1062526 · Paper page
- 10.5555/311445 · doi:10.5555/311445
- 10.5555/2820518.2820559 · doi:10.5555/2820518.2820559
- 10.5555/998675.999460 · doi:10.5555/998675.999460
Cited by
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Automating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549081 · Cited by 196
VulRepair: a T5-based automated software vulnerability repair
Fu, M., Tantithamthavorn, C., Le, T., Nguyen, V., & Phung, D. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549098 · Cited by 207
Neuro-Symbolic Program SynthesisPreprint
Parisotto, E., Mohamed, A., Singh, R., Li, L. C., Zhou, D., & Kohli, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01855 · Cited by 273
ChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
Learning to mine aligned code and natural language pairs from stack overflow
Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196408 · Cited by 218
“…researchers to expand from models of source code and natural language (NL) created using hand-crafted rules, which have a long history [23], to data-driven models that have proven flexible, relatively easy-to-create, and often more effective than corresponding hand-crafted precursors [13, 27].”
BackgroundCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
Jackson, I., Jesus Saenz, M., & Ivanov, D. (2024) · International Journal of Production Research · doi:10.1080/00207543.2023.2276811 · Cited by 173
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639180 · Cited by 267
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150
Self-citationEffective Program Debloating via Reinforcement Learning
Heo, K., Lee, W., Pashakhanloo, P., & Naik, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243838 · Cited by 150
The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring
Aniche, M. F., Maziero, E. G., Durelli, R. S., & Durelli, V. H. S. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3021736 · Cited by 155
Large Language Models and Simple, Stupid Bugs
Jesse, K., Ahmed, T., Dévanbu, P., & Morgan, E. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00082 · Cited by 146
NatGen: generative pre-training by “naturalizing” source code
Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P. T., & Ray, B. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549162 · Cited by 105
CodePlan: Repository-level Coding using LLMs and Planning
Bairi, R., Sonwane, A., Kanade, A., C, V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., & Shet, S. (2023) · arXiv · doi:10.48550/arxiv.2309.12499 · Cited by 92
CodePlan: Repository-Level Coding using LLMs and Planning
Bairi, R., Sonwane, A., Kanade, A., C., V. D., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., & Shet, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643757 · Cited by 92
A Survey of Neural Code Intelligence: Paradigms, Advances and BeyondPreprint
Sun, Q., Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.14734 · Cited by 105
Jin, X., Pei, K., Won, J. Y., & Lin, Z. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560612 · Cited by 133
Meske, C., Hermanns, T., Weiden, E. V. D., Loser, K., & Berger, T. (2025) · IEEE Access · doi:10.1109/access.2025.3645466 · Cited by 100
Effects of Variable Names on Comprehension: An Empirical Study
Avidan, E., & Feitelson, D. G. (2017) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2017.27 · Cited by 88
EditSum: A Retrieve-and-Edit Framework for Source Code Summarization
Li, J., Li, Y., Li, G., Hu, X., Xia, X., & Jin, Z. G. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678724 · Cited by 72
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Retrieve and refine: exemplar-based neural comment generation
Wei, B., Li, Y., Li, G., Xia, X., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416578 · Cited by 69
Automated detection of password leakage from public GitHub repositories
Feng, R., Yan, Z., Peng, S., & Zhang, Y. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510150 · Cited by 72
LAMD: Context-Driven Android Malware Detection and Classification with LLMs
Qian, X., Zheng, X., He, Y., Yang, S., & Cavallaro, L. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00017 · Cited by 66
AI-Driven Development Is Here: Should You Worry?
Ernst, N. A., & Bavota, G. (2022) · IEEE Software · doi:10.1109/ms.2021.3133805 · Cited by 62
Ulfsnes, R., Moe, N. B., Stray, V., & Skarpen, M. (2024) · Generative AI for Effective Software Development · doi:10.1007/978-3-031-55642-5_10 · Cited by 62
Who is using AI to code? Global diffusion and impact of generative AI
Daniotti, S., Wachs, J., Feng, X., & Neffke, F. M. H. (2026) · Science · doi:10.1126/science.adz9311 · Cited by 95
TacTok: semantics-aware proof synthesis
First, E., Brun, Y., & Guha, A. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428299 · Cited by 59
Retrieve and Refine: Exemplar-Based Neural Comment Generation
Wei, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00152 · Cited by 59
M2TS: multi-scale multi-modal approach based on transformer for source code summarization
Gao, Y., & Lyu, C. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527907 · Cited by 66
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 81
Feitelson, D. G., Mizrahi, A., Noy, N., Shabat, A. B., Eliyahu, O., & Sheffer, R. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2976920 · Cited by 63
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 61
ExploitGen: Template-augmented exploit code generation based on CodeBERT
Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., & Chen, T. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111577 · Cited by 64
Aligning the Objective of LLM-Based Program Repair
Xu, J., Fu, Y., Tan, S. H., & He, P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00169 · Cited by 51
Diversity-driven automated formal verification
First, E., & Brun, Y. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510138 · Cited by 48
What is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47
Making the most of small Software Engineering datasets with modern machine learning
Prenner, J. A., & Robbes, R. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3135465 · Cited by 64
PYEVOLVE: Automating Frequent Code Changes in Python ML Systems
Dilhara, M., Dig, D., & Ketkar, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00091 · Cited by 46
Building Language Models for Text with Named Entities
Parvez, M. R., Chakraborty, S., Ray, B., & Chang, K.-W. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1221 · Cited by 43
Large Language Models are Not Models of Natural Language: They are Corpus Models
Veres, C. (2022) · IEEE Access · doi:10.1109/access.2022.3182505 · Cited by 49
Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers
Shi, Y., Zhang, H., Wan, C., & Gu, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00005 · Cited by 67
Automatic source code summarization with graph attention networks
Zhou, Y., Shen, J., Zhang, X., Yang, W., Han, T., & Chen, T. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111257 · Cited by 40
Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries
Al-Kaswan, A., Ahmed, T., Izadi, M., Sawant, A. A., Dévanbu, P., & Arie Van Deursen, A. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00033 · Cited by 72
Survey of Code Search Based on Deep Learning
Xie, Y., Lin, J., Dong, H., Zhang, L., & Wu, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628161 · Cited by 56
Correctness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 48
Discovering repetitive code changes in python ML systems
Dilhara, M., Ketkar, A., Sannidhi, N., & Dig, D. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510225 · Cited by 39
What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
Gu, W., Chen, J., Wang, Y., Jiang, T., Li, X., Liu, M., Liu, X., Ma, Y., & Zheng, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773224 · Cited by 38
Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach
Ampel, B., Samtani, S., Zhu, H., Ullman, S., & Chen, H. (2020) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi49825.2020.9280548 · Cited by 38
Code-centric learning-based just-in-time vulnerability detection
Nguyen, S., Nguyen, T.-T., Vu, T. T., Do, T.-D., Ngo, K.-T., & Vo, H. D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112014 · Cited by 42
Learning to recommend method names with global context
Liu, F., Li, G., Fu, Z., Lu, S., Hao, Y., & Jin, Z. G. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510154 · Cited by 37
Software as storytelling: A systematic literature review
Ciancarini, P., Farina, M., Okonicha, O., Smirnova, M., & Succi, G. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100517 · Cited by 41
Learning to recognize actionable static code warnings (is intrinsically easy)
Yang, X., Chen, J., Yedida, R., Yu, Z., & Menzies, T. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09948-6 · Cited by 42
Design pattern recognition: a study of large language models
Pandey, S. K., Chand, S., Horkoff, J., Staron, M., Ochodek, M., & Durisic, D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10625-1 · Cited by 36
López, J. A. H., Weyssow, M., Cuadrado, J. S., & Sahraoui, H. A. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556900 · Cited by 55
Jain, R., Gervasoni, N., Ndhlovu, M., & Rawat, S. (2023) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3578527.3578530 · Cited by 35
How Important Are Good Method Names in Neural Code Generation? A Model Robustness Perspective
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., & Chen, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3630010 · Cited by 44
Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example
Dilhara, M., Bellur, A., Bryksin, T., & Dig, D. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643755 · Cited by 32
(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Bin2vec: learning representations of binary executable programs for security tasks
Arakelyan, S., Arasteh, S., Hauser, C., Kline, E., & Galstyan, A. (2021) · Cybersecurity · doi:10.1186/s42400-021-00088-4 · Cited by 31
Cross-project defect prediction based on G-LSTM model
Xing, Y., Qian, X., Guan, Y., Yang, B., & Zhang, Y. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.04.039 · Cited by 31
A Comprehensive Study on Large Language Models for Mutation Testing
Wang, B., Chen, M., Deng, M., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3805038 · Cited by 35
Navigating Complexity in Software Engineering: A Prototype for Comparing GPT-n Solutions
Treude, C. (2023) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse59190.2023.00008 · Cited by 30
SrcMarker: Dual-Channel Source Code Watermarking via Scalable Code Transformations
Yang, B., Li, W., Xiang, L., & Li, B. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00097 · Cited by 42
AutoPruner: transformer-based call graph pruning
Le-Cong, T., Kang, H. J., Nguyen, T. G., Haryono, S. A., Lo, D., Le, X.-B. D., & Huynh, Q. T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549175 · Cited by 32
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 31
Bridging expert knowledge with deep learning techniques for just-in-time defect prediction
Zhou, X., Han, D., & Lo, D. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10591-0 · Cited by 29
A Software Defect Prediction Method Based on Program Semantic Feature Mining
Yao, W., Shafiq, M., Lin, X., & Yu, X. (2023) · Electronics · doi:10.3390/electronics12071546 · Cited by 24
Revisiting Unnaturalness for Automated Program Repair in the Era of Large Language Models
Yang, A. Z. H., Kolak, S., Hellendoorn, V. J., Martins, R., & Goues, C. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00089 · Cited by 24
From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering
Ullrich, J., Koch, M. A., & Vogelsang, A. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00032 · Cited by 23
Enhancing Source Code Refactoring Detection with Explanations from Commit Messages
Krasniqi, R., & Cleland‐Huang, J. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054816 · Cited by 22
On the cross-modal transfer from natural language to code through adapter modules
Goel, D., Grover, R., & Fard, F. H. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527892 · Cited by 30
Studying the characteristics of AIOps projects on GitHub
Aghili, R., Li, H., & Khomh, F. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10382-z · Cited by 22
FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion
Guo, Q., Li, X., Xie, X., Liu, S., Tang, Z. J., Feng, R., Wang, J., Ge, J., & Bu, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652130 · Cited by 28
Mining Python fix patterns via analyzing fine-grained source code changes
Yang, Y., He, T., Feng, Y., Liu, S., & Xu, B. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10087-1 · Cited by 21
Inferring and applying type changes
Ketkar, A., Smirnov, O., Tsantalis, N., Dig, D., & Bryksin, T. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510115 · Cited by 21
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
van Dam, T., Izadi, M., & Arie Van Deursen, A. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00035 · Cited by 25
Automatic Static Vulnerability Detection for Machine Learning Libraries: Are We There Yet?
Harzevili, N. S., Shin, J., Wang, J., Wang, S., & Nagappan, N. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00042 · Cited by 25
Syntax-aware on-the-fly code completion
Takerngsaksiri, W., Tantithamthavorn, C. K., & Li, Y.-F. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107336 · Cited by 27
Exploring GNN based program embedding technologies for binary related tasks
Guo, Y., Li, P., Luo, Y., Wang, X., & Wang, Z. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527900 · Cited by 19
Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions
Sun, K., Kuang, H., Baltes, S., Zhou, X., Zhang, H., Ma, X., Rong, G., Shao, D., & Treude, C. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3688237 · Cited by 24
DAISY: Dynamic-Analysis-Induced Source Discovery for Sensitive Data
Zhang, X., Heaps, J., Slavin, R., Niu, J., Breaux, T. D., & Wang, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569936 · Cited by 18
Dai, P., Li, Y., Wang, Y., Jin, D., & Gong, Y. (2023) · Mathematics · doi:10.3390/math11020315 · Cited by 18
Rethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 23
Patch Correctness Assessment: A Survey
Fei, Z., Ge, J., Li, C., Wang, T., Li, Y., Zhang, H., Huang, L., & Luo, B. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702972 · Cited by 25
Machine Learning Approaches to Code Similarity Measurement: A Systematic Review
Zhang, Z., & Saber, T. (2025) · IEEE Access · doi:10.1109/access.2025.3553392 · Cited by 18
Show Me Your Code! Kill Code Poisoning: A Lightweight Method Based on Code Naturalness
Sun, W., Chen, Y., Yuan, M., Fang, C., Chen, Z., Wang, C., Liu, Y., Xu, B., & Chen, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00247 · Cited by 26
Chen, T., Rami Bahsoon, A., & Yao, X. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2985293 · Cited by 18
Gated Homogeneous Fusion Networks With Jointed Feature Extraction for Defect Prediction
Fang, D., Liu, S., & Ai, L. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3165115 · Cited by 18
Fine-grained code-comment semantic interaction analysis
Geng, M., Wang, S., Dong, D., Gu, S., Peng, F., Ruan, W., & Liao, X. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527887 · Cited by 17
Rambo: Enhancing RAG-based repository-level method body completion
Bui, T.-D., Luu-Van, D.-T., Nguyen, T.-P., Nguyen, T.-T., Vo, H. D., & Nguyen, S. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108322 · Cited by 17
LSTM-based argument recommendation for non-API methods
Li, G., Liu, H., Li, G., Shen, S., & Tang, H. (2020) · Science China Information Sciences · doi:10.1007/s11432-019-2830-8 · Cited by 17
ARist: An effective API argument recommendation approach
Nguyen, S., Tran, C., Tran, K. T., Nguyen, T. M., Nguyen, T.-T., Ngo, K.-T., & Vo, H. D. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111786 · Cited by 17
Saberi, I., Fard, F. H., & Chen, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10457-5 · Cited by 22
DeepClone: Modeling Clones to Generate Code Predictions
Hammad, M., Babur, Ö., Basit, H. A., & van den Brand, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64694-3_9 · Cited by 17
Specification-driven code translation powered by Large Language Models: How far are we?
Saha, S. K., Rabbi, F., Pham, T. M. T., Wang, S., & Yang, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112964 · Cited by 18
Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing
Le-Cong, T., Nguyen, T.-D., Le, X.-B. D., & Murray, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716167 · Cited by 16
RepoVis: Visual Overviews and Full-Text Search in Software Repositories
Feiner, J., & Andrews, K. (2018) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2018.00009 · Cited by 16
Metadata
- Authors
- Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P.
- Affiliations
- University of Alberta (CA)
- University College London (GB)
- University of California, Davis (US)
- UC Davis, CA
- Venue
- Communications of the ACM
- Published
- 2016-04-26
- Volume
- 59
- Issue
- 5
- Pages
- 122-131
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2344444819
- Cited by
- 307
- References
- 56
- Citations by year
- 2026: 34 · 2025: 58 · 2024: 48 · 2023: 45 · 2022: 66 · 2021: 26 · 2020: 16 · 2019: 5 · 2018: 6 · 2017: 2 · 2016: 1
- FWCI
- 10.42
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/J017515/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.