On the naturalness of software
The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations - and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The IntelliCode Compose system uses the term click through rate (CTR) for this and reports a value of 10% in online trials.20”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [20]
In the citing paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [11]
Citations
Cites
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Statistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 871
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Mining version histories to guide software changes
Zimmermann, T., Weibgerber, P., Diehl, S., & Zeller, A. (2004) · Proceedings. 26th International Conference on Software Engineering · doi:10.1109/icse.2004.1317478 · Cited by 322
Statistical language model adaptation: review and perspectives
Bellegarda, J. R. (2004) · Speech Communication · doi:10.1016/j.specom.2003.08.002 · Cited by 297
Cited by
Items 801–834 of 834
Introduction Of Generative Ai On Software DevelopmentPreprint
Gaidhane, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6628118
Obfuscation by Simplification: How LLMs Transform Code Through Structural ExpansionPreprint
Tomasi, L. D., Sipio, C. D., Marco, A. D., & Nguyen, P. T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7211169
From Command to Conversation: Integrating Agentic AI Into Command-Line Interfaces
Shrivastava, M. (2026) · Computer · doi:10.1109/mc.2026.3673219
Evaluating package-level scoping strategies for repository-level code completion in Pharo
AbedelKader, O., Ducasse, S., Polito, G., Zaitsev, O., & Robbes, R. (2026) · Journal of Computer Languages · doi:10.1016/j.cola.2026.101412
Lepp, M., & Kaimre, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16318
A Survey of Symbol Name Recovery in Software Reverse Engineering
Fan, H., Wu, J., He, X., Feng, Y., Yang, Y., Xu, B., & Shi, Q. (2026) · ACM Computing Surveys · doi:10.1145/3844948
Shen, X., Xi, S.-Z., Du, Y., & Li, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.26388
Duan, B., Lin, J., Kim, D. S., & Yang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.26234
Towards an Evaluation Methodology for AI Code Assistants
Lekarakos, A. I., Koutsomitropoulos, D. A., & Tsolis, D. K. (2026) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa71656.2026.11705481
ming, X., & Dao, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7538299
Automated Program Repair for UI-Centric Android Bugs: How Far Are We?
Mahmud, J., Pandey, S., Silva, N. D., Dipongkor, A. K., Wu, J., Chaparro, O., Fazzini, M., & Moran, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832272
Crowdsourced documentation and code snippets
Geremia, S. (2020) · Archivio Istituzionale della Ricerca - Università degli Studi del Molise
Martins, S. (2024) · International Journal of Next-Generation Engineering and Technology
Kotmir, V. B. (2024) · Advances in Nonlinear Variational Inequalities
Phung, T.-N., & Nguyen, V. T. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108363
A Comparative Study of Rule-Based and LLM-Based IaC Security Misconfiguration Detection in DevOps
Bari, B. (2026) · UTUPub (University of Turku)
A Hybrid AI-Driven Framework for Automated Code Review and Defect Detection
Nagar, S., Chauhan, P., Khan, S., & Kumar, A. (2026) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic70037.2026.11717358
MAP4CS: A Multi-dimensional Data Pruning Framework for Efficient Code Retriever Fine-tuningPreprint
Chen, Y., Liu, M., Ou, G., Zhang, Z., Li, Z., Wang, Y., & Zheng, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.11727
Marchezan, L., Lefeuvre, R., Delcourt, K., Combemale, B., & Sahraoui, H. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.24093
A Critical Review of Cognitive Complexity Metrics and Tools: Gaps and Opportunities for ERP Systems
Wijesinghe, S. W. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-34621-6_32
Studying proof data with proof theory: the case of AI-generated mathematicsPreprint
Aryal, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.25357
Precedent or Intent? What Coding Agents Follow When Evolving Existing CodePreprint
Zhu, H., Xu, G., Yang, Z., Wang, J., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.38244
PyReprism: A Shared Lexical Foundation for Polyglot Code AnalysisPreprint
Ogenrwot, D., Cervantes, J. G. D., Sserunjogi, R., & Businge, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.39224
Program Bindings Inside Code ModelsPreprint
Rajput, P. K., Song, Y., Klein, J., & Bissyandé, T. F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.39489
Diagnosing and Correcting Geometric Bias in Diversity-Driven Code Instruction Data Selection
Zhang, K., Huang, Z., & Cheng, G. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3841133
Orchestrated Multi-Agent and LLM-Based Reasoning for Static Binary Code Analysis
Alshaali, M., Tu, W., & Hacid, H. (2026) · IEEE Access · doi:10.1109/access.2026.3750045
OpCode-Stats: A Statistical Analysis of Compiled Program Instruction Sequences
Aslyan, A., & Harutyunyan, A. N. (2026) · doi:10.1109/codassca69992.2026.00053
Software Defect Prediction Model Fusing File and Program Graph Representation
Zhou, X., Zhu, X., Wang, Y., Tian, S., Wang, J., Lai, X., & Li, X. (2026) · Software Testing Verification and Reliability · doi:10.1002/stvr.70032
Autonomous AI Agents Systematically Diverge From Human Preferences in Proof FormalizationPreprint
Gyevnár, B., & DeDeo, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.06995
Hierarchical Bayesian Defect Localization under Test and Inspection Constraints
Gong, H. (2026) · Journal of Computational Methods in Engineering Applications · doi:10.62836/jcmea.v6i1.0004
Graph-Evidence Repository Repair: Risk-Calibrated Retrieval and Patch Gating on SWE-Bench
Gong, H. (2026) · Journal of Information Technology and Policy · doi:10.62836/jitp.v4i1.0003
Wang, Y., Yang, C., Hou, S., Xu, X., Cui, B., 蒋严冰, & Yang, T. (2026) · Data Science and Engineering · doi:10.1007/s41019-026-00371-9
KG-Commit: A Dynamic Knowledge Graph for Online Just-in-Time Software Defect Prediction
Hesamolhokama, M., Aghbash, M. S. B., Rohani, B., Fazli, M., & Habibi, J. (2026) · arXiv · doi:10.48550/arxiv.2609.06272
Metadata
- Authors
- Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- The University of Texas at Dallas (US)
- Venue
- 2012 34th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 837-847
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W3146720657
- Cited by
- 425
- References
- 39
- Citations by year
- 2026: 19 · 2025: 24 · 2024: 36 · 2023: 44 · 2022: 49 · 2021: 50 · 2020: 48 · 2019: 46 · 2018: 22 · 2017: 35 · 2016: 13 · 2015: 23 · 2014: 2 · 2013: 14
- FWCI
- 9.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support, natural language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.