On the naturalness of software
The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural , in the sense that it is created by humans at work, with all the attendant constraints and limitations---and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether (a) code can be usefully modeled by statistical language models and (b) such models can be leveraged to support software engineers. Using the widely adopted n -gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very regular, and, in fact, even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
On The Language Instinct: (412952005-009)
Pinker, S., & American Psychological Association (1994) · PsycEXTRA Dataset · doi:10.1037/e412952005-009 · Cited by 5,646
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Knuth, D. E. (1984) · The Computer Journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,467
Statistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · cited 1×
“Summarization and Concern Location A line of work on code summarization 9, 47, 48 uses semantic properties inferred by static analysis, rather than the “natural” regularities of software corpora capturable in statistical models; it is complementary to ours: properties derived by static analysis (as…”
MethodMining Version Histories to Guide Software Changes
Zimmermann, T., Member, S., Weißgerber, P., Diehl, S., Zeller, A., & Ieee Computer Society (2004) · Cited by 494
Building a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 489
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Research works printed but not linked: 6
- 10.5555/2819009.2819143 · doi:10.5555/2819009.2819143
- 10.5555/1734086 · doi:10.5555/1734086
- 10.1145/1062455.1062526 · doi:10.1145/1062455.1062526 · Paper page
- 10.5555/311445 · doi:10.5555/311445
- 10.5555/2820518.2820559 · doi:10.5555/2820518.2820559
- 10.5555/998675.999460 · doi:10.5555/998675.999460
Cited by
Items 301–330 of 330
Effective Vulnerability Detection over Code Token Graph: A GCN with Score Gate Based Approach
Zou, N., Li, N., Zhang, J., Wang, X., Hong, L., & Jia, T. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00025
TriGraph: A Probabilistic Subgraph-Based Model for Visual Code Completion in Pure Data
Islam, A., & Hindle, A. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00110 · Cited by 1
Self-citationVulGTDA: A Software Vulnerability Detection Method via Graph Transformer and Domain Adaptation
Cui, H., Zhang, C., & Cai, F. (2025) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice64954.2025.11063870
Towards understanding the security issues of Python programs
Fan, H., Liu, D., Wu, J., Feng, Y., Shi, Q., & Xu, B. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755886
Enhancing Binary Security Analysis Through Pre-Trained Semantic and Structural Feature Matching
Chen, Y., Dai, W., Deng, Y., Bao, L., & Xu, G. (2025) · Applied Sciences · doi:10.3390/app152111610
Improving code completion efficiency through grouped attention
Yin, Y., Liu, J., Liu, Y., & Jia, D. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf117
Chen, J., Li, Z., Xu, J., Cai, S., Sun, J., Sosu, R. N. A., & Chen, J. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf126
Tuape, M., Kasurinen, J., Najjemba, M. T., & Islam, A. D. (2025) · International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) · doi:10.1109/iceccme64568.2025.11277750
Improving the Reliability of AI Infrastructure Software with Data-Driven Software Analytics
Shiri Harzevili, N. (2025) · York University Digital Library (York University)
Evaluation of Foundation Models for Infrastructure-as-Code Automation on Amazon Bedrock
Terry, B., Shi, Q., Yang, Z., Haidar, A., & Shao, S. (2025) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars67163.2025.11337933
Modeling Transfer Learning for Efficient Code Reuse in Large-Scale Software Development
Moltu, J. L., Muge, P. T., & BASSI, J. Y. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150100008
MTL-CR: A Multitask Learning Approach for Code Representation
Yu, D., Mao, F., Du, X., & Bai, Y. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00030
Li, R., Mo, R., Zeng, Y., Liu, Z., Li, Z., & Ma, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3800582 · Cited by 4
Wang, Y., Guo, S., Tao, Z., Li, H., Tan, J., Hao, L., Zhu, F., Chen, X., & Youssef, M. (2026) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2026.3676742 · Cited by 1
MaliVD: Source Code Vulnerability Localization via Attention-Based Multi-Modal Learning
Dai, E., Wang, S., Xiao, X., Li, Q., Wen, S., & Zhu, T. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3681191
Muñoz-Babiano, F., Lamo, P., & Alonso, R. S. (2026) · ADCAIJ ADVANCES IN DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE JOURNAL · doi:10.14201/adcaij.32926
Test-free fault localization using large language models and a Transformer-Mamba hybrid architecture
Huang, Y., Lei, Y., & Xie, H. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108165
AdaptGen: A Problem-Adaptive Solution Template Generation Technique for Online Programming Platforms
Zhang, G. T., Zou, W., Zhang, X., Zhang, J., & Xuan, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3692192
A Lightweight Defense Against Code Poisoning Attacks Based on Code Naturalness
Yuan, M., Sun, W., Chen, Y., Fang, C., Chen, Z., Lv, P., Guo, W., Liu, Y., Xu, B., Chen, Z. L., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3705037
Transpilation Using Recursive Rewrite Rules: From Legacy to Maintainable Code
Albers, T., Hegge, J., van de Laar, P., Brouwers, N., Nelissen, P., Alberts, W., Azis, G., Baan, T., Handoko, D., & van der Linden, Q. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00031
When AI Coding Assistants Leak Training Data: A Study of LLM Memorization in Code Generation
Cheng, X., Yao, K., Nie, P., & Shang, W. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814902 · Cited by 1
Evaluating Large Language Models for C/C++ Software Composition Analysis
Ai, Y., Zhao, Y., Cheng, Y., & Liu, Q. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581685
What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study
Cotroneo, D., Rosa, G. D., Improta, C., & Varriale, B. G. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00027 · Cited by 1
Defending neural code understanding models by eliminating backdoors
Cheng, W., Zhou, Y., Yang, G., Zhang, X., Yang, W., & Chen, T. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00645-6
Jiang, Y., Zhou, Q., Wu, D., Zhao, S., & Lin, C. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01240-4
BiVCoder: A Multi-Agent Framework for Code Generation via Bidirectional Code-Test Diagnosis
Li, X., Dong, J., Shi, W., Lü, W., & Du, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818107
RL–LLMfuzzer: reinforcement learning–guided dual–model differential fuzzing for compiler testing
Li, D., Li, Y., Zhao, B., Xu, J., Yao, J., Wang, N., & Liu, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01115-5
Qiu, Y., Wang, S., Qin, Y., Geng, M., & Dong, D. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837454
StackLoc: A CodeBERT-BiLSTM framework for stack overflow vulnerability detection and localization
Zhai, J., Qi, Z., Miao, H., Xu, T., & Yang, H. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09791-2
Obrębska-Woźniczka, A. (2026) · Językoznawstwo · doi:10.25312/j.10411
Metadata
- Authors
- Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P.
- Affiliations
- University of Alberta (CA)
- University College London (GB)
- University of California, Davis (US)
- UC Davis, CA
- Venue
- Communications of the ACM
- Published
- 2016-04-26
- Volume
- 59
- Issue
- 5
- Pages
- 122-131
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2344444819
- Cited by
- 307
- References
- 56
- Citations by year
- 2026: 34 · 2025: 58 · 2024: 48 · 2023: 45 · 2022: 66 · 2021: 26 · 2020: 16 · 2019: 5 · 2018: 6 · 2017: 2 · 2016: 1
- FWCI
- 10.42
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/J017515/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.