On the naturalness of software
The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations - and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The IntelliCode Compose system uses the term click through rate (CTR) for this and reports a value of 10% in online trials.20”
From Measuring GitHub Copilot's Impact on Productivity (2024) · cites this paper as [20]
In the citing paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [11]
Citations
Cites
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Statistical Machine Translation
Koehn, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815829 · Cited by 871
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Mining version histories to guide software changes
Zimmermann, T., Weibgerber, P., Diehl, S., & Zeller, A. (2004) · Proceedings. 26th International Conference on Software Engineering · doi:10.1109/icse.2004.1317478 · Cited by 322
Statistical language model adaptation: review and perspectives
Bellegarda, J. R. (2004) · Speech Communication · doi:10.1016/j.specom.2003.08.002 · Cited by 297
Cited by
Items 701–800 of 834
Replacements and Replaceables: Making the Case for Code VariantsPreprint
Vinayakarao, V., Sondhi, D., Keswani, S., Purandare, R., & Sarma, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03844
Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04137
de Sousa, R. R. (2020) · Brazilian Journal of Development · doi:10.34117/bjdv6n7-361
Automatische Bewertung von Codequalität innerhalb eines Code Reviews
Lemke, S. L. (2020) · ePublications (Technical University of Cologne)
Beyond Similar Code: Leveraging Social Coding Websites
Yang, D. (2019) · eScholarship (California Digital Library)
An Improved k-Means Methods for Cloud Computing based Coding Big Data Environment
Li, X., Lin, R., & Xu, H. (2020) · International Conference on Computer Communication and Network Security (CCNS) · doi:10.1109/ccns50731.2020.00044
Chen, J., Shen, X., & Menzies, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02817
Derin Öğrenme ile Otomatik Hata Triyajlama: Bir Replikasyon Çalışması
Tüzün, E., DOĞAN, E., & ÇETİN, A. (2021) · European Journal of Science and Technology · doi:10.31590/ejosat.781341
Scalable deep learning for bug detection
Karampatsis, R.-M. (2021) · ERA · doi:10.7488/era/1074
Classification Model Based Automatic Static Evaluation Method for C Programs
Shu, X., & Fangyao, G. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-89698-0_47
Code Generation with Hybrid of Structural and Semantic Features Retrieval
Yang, K., Yu, H., Fan, G., Huang, Z., & Zhou, Z. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500267
Toward an accurate method renaming approach via structural and lexical analyses
Luo, J., Zhang, J., Huang, Z., Xu, Y., & Sun, C. (2022) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2100470
Efficient Information Retrieval for Software Bug Localization
Khatiwada, S. (2022) · Louisiana State University · doi:10.31390/gradschool_dissertations.5769
Phrase2Set: Phrase-to-Set Machine Translation and Its Software Engineering Applications
Nguyen, T. V., Yadavally, A., & Nguyen, T. N. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00068
Machine Learning for Programming Language Processing
Alon, U. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6730560
Exploring the Impact of Balanced and Imbalanced Learning in Source Code Suggestion
Hussain, Y., Huang, Z., Zhou, Y., & Khan, I. A. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500589
Feature Location in Models (FLiM): Design Time and Runtime
Arcega, L., Font, J., Haugen, Ø., & Cetina, C. (2022) · Springer eBooks · doi:10.1007/978-3-031-11686-5_4
Development of a Program Code Review System Using Machine Learning Methods
Moshkin, V., & Andreev, I. A. (2023) · Studies in systems, decision and control · doi:10.1007/978-3-031-22938-1_5
Natural Language-Guided Programming User Study
Heyman, G., Huysegems, R., Justen, P., & Cutsem, T. V. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5384767
A Novel Code Classification based on Recurrent Nerual Network and Multi-Head Attention
Zhao, L., Huang, F., Kang, L., Chang, J., Liu, X., & Fan, Y. (2023) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst60924.2023.10502705
Modeling source code in bimodal for program comprehension
Wen, D., Zhang, X., Diao, Y., Zhao, Z., Jiang, H., & Lin, H. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09498-0
Cognitive Predictors of Student Success in the Acquisition of Computer Programming
Egan, S. C. (2024) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2024-15891
Zhang, S., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66459-5_1
Profiling Software Developers: Fingerprints and textual abstractions of software process smellsPreprint
Caldeira, J., Abreu, F. B. E., Cardoso, J., Ribeiro, R., & Werner, C. M. L. (2024) · doi:10.22541/au.172493465.54289190/v1
Demystifying and Assessing Code Understandability in Java DecompilationPreprint
Qin, R., Xiong, Y., Lu, Y., & Pan, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.20343
Contextual Augmented Multi-Model Programming (CAMP): A Hybrid Local-Cloud Copilot FrameworkPreprint
Wang, Y., Guo, S., & Tan, C. W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.15285
Lost in Transmission: An Information-Theoretic Account of Unsupervised Software Traceability
Rodríguez-Cárdenas, D., Fecko, L., Poshyvanyk, D., Palacio, D. N., & Moran, K. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00129
Checking Test Suite Efficacy Through Dual-Channel Techniques
Petrescu, C. C., Smith, S., Butler, A., & Dash, S. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80889-0_19
Structure‐Aware Representation Learning for Effective Performance Prediction
Ramadan, T., Pinnow, N., Phelps, C., Thiagarajan, J. J., & Islam, T. Z. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70046
Pauzi, Z., Sas, C., & Capiluppi, A. (2025) · doi:10.22541/au.174704970.03040102/v1
MODELOS DE RACIOCÍNIO E ARQUITETURA EXTENSÍVEL PARA PESQUISA INTELIGENTE
Júnior, H. C. P. (2025) · Congresso Internacional IDEA: Inovação, Diálogo e Experiências na Aprendizagem (IDEA) · doi:10.29327/9786527213314.1086647
Terminal Lucidity: Envisioning the Future of the Terminal
MacInnis, M., Baysal, O., & Lanza, M. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00042
Natarajan, G., Elango, E., Cyriac, R., & Muthusamy, S. (2025) · Advances in computers · doi:10.1016/bs.adcom.2025.06.016
Program Trace Optimization for Language Model Search: the Abstraction and Reasoning Corpus case
Moraglio, A., & Tonda, A. P. (2025) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3712255.3726554
Towards a Science of Causal Interpretability in Deep Learning for Software EngineeringPreprint
Palacio, D. N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.15023
AI-powered Document Chatbot for data retrieval
Raj, M. K., & Srinivasan, R. (2025) · Atlantis Highlights in Intelligent Systems/Atlantis highlights in intelligent systems · doi:10.2991/978-94-6463-866-0_55
Fault Localisation and Repair for DL Systems: An Empirical Study with LLMsPreprint
Kim, J.-H., Humbatova, N., Jahangirova, G., Yoo, S., & Tonella, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.03396
AI-Augmented Software Architecture: Autonomous Refactoring with Design Pattern Awareness
Konakanchi, M. S. K. (2025) · International Conference on Recent Trends in Computer Science and Information Technology · doi:10.63282/3050-9246/icrtcsit-102
Embedding-Based Semantic Tracing: Mapping Concepts to Domains in Software ArtifactsPreprint
Pauzi, Z., Sas, C., & Capiluppi, A. (2025) · doi:10.22541/au.176374712.28513607/v1
AI-Powered Code Completion with LSTM Networks
Bani-Salameh, H., Fattoum, Y., & Hjeela, F. A. (2025) · National Conference on Electrical and Electronics Engineering (ELECO) · doi:10.1109/iceee67194.2025.11261923
Programming with "Big Code": Lessons, Techniques and Applications
Bielik, P., Raychev, V., & Vechev, M. (2015) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000110465
Terminal Lucidity: Envisioning the Future of the TerminalPreprint
MacInnis, M., Baysal, O., & Lanza, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.13994
Mauczka, A. (2016) · reposiTUm (TU Wien) · doi:10.34726/hss.2016.38867
Automated Generic Optimization in Real time using Mutation Operators
Silva, P. M. O. C. D. (2021) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/wevr-7404
LEARNING PROGRAM EMBEDDING FROM UNLABELED SOURCE CODE
Ahmed, T. (2023) · eScholarship (California Digital Library)
TARGETED, REALISTIC AND NATURAL FAULT INJECTION : (USING BUG REPORTS AND GENERATIVE LANGUAGE MODELS)
Khanfir, A. (2023) · Open Repository and Bibliography (University of Luxembourg)
Learning Off-By-One Mistakes: An Empirical Study on Different Deep Learning Models
Sellik, H. (2020) · Research Repository (Delft University of Technology)
Gholamian, S. (2022) · UWSpace (University of Waterloo)
Investigating the Relationship Between Surprisal and Processing in Programming Languages
Dodd, N., Reese, S. J., & Morgan, E. (2024) · eScholarship (California Digital Library)
Learning Code Change Semantics for Patch Correctness Assessment in Program Repair
TIAN, H. (2023) · Open Repository and Bibliography (University of Luxembourg)
Round-Trip Translation : A New Path for Automatic Program Repair using Large Language Models
Ruiz, F. V. (2023) · DiVA (Digitala Vetenskapliga Arkivet)
A Security Requirements Management Framework for Open-Source Software Projects
Wang, W. T. (2019) · OhioLink ETD Center (Ohio Library and Information Network)
CODE-CHANGE AWARE MUTATION BASED TESTING IN CONTINUOUSLY EVOLVING SYSTEMS
Ojdanić, M. (2023) · Open Repository and Bibliography (University of Luxembourg)
オープンソースソフトウェアプロジェクトにおけるソフトウェア品質保持に関する研究
ヌリ, オ. (2024) · Institutional Repositories DataBase (IRDB)
Elastic Dynamic Binary Analysis and Applications
Qi, Z. (2024) · eScholarship (California Digital Library)
Extraction et analyse de connaissance pour la réparation automatique de bugs
Martínez, M. (2014) · theses.fr (ABES)
A Language-Parametric Modular Framework for Mining Idiomatic Code Patterns
Pham, H. S. (2019) · VUBIR (Vrije Universiteit Brussel)
Learning Assembly Language Models for Security Applications
Li, X. (2024) · eScholarship (California Digital Library)
Contextual cues for deep learning models of code
Shrivastava, D. (2023) · Papyrus : Institutional Repository (Université de Montréal) · doi:10.71781/10146
Neurosymbolic Tools for Effective Coding and Debugging
Sakkas, G. (2024) · eScholarship (California Digital Library)
Beyond Natural Language Processing: Advancing Software Engineering Tasks through Code Structure
Ding, Z. (2024) · UWSpace (University of Waterloo)
Extraction and Analysis of Knowledge for Automatic Software Repair
Martínez, M. (2014) · HAL
Learning to Predict Software Performance Changes based on Microbenchmarks
David, L. (2024) · Qucosa (Saxon State and University Library Dresden)
Enhancing Security Patch Documentation Through Generative AI
Majeri, I. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Representation Learning of Software Source Code
PIAN, W. (2025) · Open Repository and Bibliography (University of Luxembourg)
CoSrch: Syntactic and Semantic Structure-Aware Code Representation for Effective Code Search
Xu, J., He, Y., Liu, H., Lv, P., Zhang, T., & Tian, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785469
Source Code Vulnerability Detection and Interpretability with Language Models
Silveira, L., Lélis, C. A. S., Marcondes, C. A. C., & Verri, F. A. N. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-11539-3_15
Zhu, C., Chen, J., & Liu, Y. (2024) · Journal of Advanced Computing Systems · doi:10.69987/jacs.2024.41206
Narrowing the Complexity Gap in the Evaluation of Large Language ModelsPreprint
Chen, Y., Liu, S., & Jabbarvand, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.18928
Tekoälyavusteinen ohjelmistokehitys: GitHub Copilotin arviointi ohjelmistokehittäjän työkaluna
Kuusisto, H. (2024) · Tampere University Institutional Repository (Tampere University)
Generative AI in Software Engineering
H, A. C. (2026) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2026.77834
From Verification to Herding: Exploiting Software's Sparsity of InfluencePreprint
Menzies, T., & Ganguly, K. K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.10478
Ruohonen, J. (2026) · UTUPub (University of Turku)
How challenging it is to identify real code authors: an empirical study
Gong, S., & Zhong, H. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10819-1
Representation Learning for Program Analysis, Testing and Repair
LOYOLA, H. P. S. (2017) · Institutional Repositories DataBase (IRDB) · doi:10.15083/00077638
A Version-Aware Metric for Measuring Innovation in Software Code Patterns
Rinas, A. M., & Wickramasinghe, B. M. G. S. T. S. K. (2026) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc68737.2026.11453625
Teegala, R. (2024) · International Journal of Scientific Research in Science Engineering and Technology · doi:10.32628/ijsrset24118043
Al-Powered Codebase Analysis and Explanation Tool
(2026) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets93516
MEASURING SOFTWARE DEVELOPER PRODUCTIVITY: REVIEW OF APPROACHES AND THEIR LIMITATIONS
Handzhiev, V. (2026) · E-Journal VFU · doi:10.53606/evfu.25.620-628
Individuazione automatica di Design Pattern all’interno del codice
FRANCO, A. (2026) · Electronic Theses and Dissertations Repository (University of Pisa)
CodeMitra: A Competitive Programming and Coding Education Platform for Indian College Students
Rathi, A. N. (2026) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2026.80861
Yao, W., Jiang, M., & Zhou, Y. (2026) · Applied Sciences · doi:10.3390/app16094416
ANALYSIS OF THE DYNAMICS OF THE POPULARITY OF PROGRAMMING LANGUAGES ACCORDING TO OPEN INDEX DATA
КУЧМА, Ю., Yanko, A. S., Rubin, E., & ПОЛІНОВСЬКИЙ, В. (2026) · Collection of Scientific Publications NUS · doi:10.15589/znp2026.1(503).2.2
Towards Accurate Detection of Security-Related Misconfigurations in IaC Scripts
War, A., Rawass, A. A., Kaboré, A. K., Samhi, J., Olatunji, I. E., Klein, J., & Bissyandé, T. F. (2026) · Open Repository and Bibliography (University of Luxembourg)
Quantifying SZZ-induced evaluation bias in just-in-time defect prediction
Bludau, P., & Pretschner, A. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09796-x
The Security Budget of Code-LLM Prompt Hardening: Provable Limits Under Pass-Only AcceptancePreprint
Tai, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03308
From Is to How: A New Framework for AI Coding AgentCapability AssessmentPreprint
Sun, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6880639
A Lightweight Defense Against Code Poisoning Attacks Based on Code Naturalness
Yuan, M., Sun, W., Chen, Y., Fang, C., Chen, Z., Lv, P., Guo, W., Liu, Y., Xu, B., Chen, Z. L., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3705037
Streamlining Repository Tasks with Effective Snippet Retrieval
Xu, T., Li, C., Xu, Z., Jiang, Y., Yao, Y., Zhu, X., Xu, F., Di, P., Xu, C., & Su, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821433
Self-citationAutomated Code Formatting Framework Using Hybrid N-gram and LSTM ModelsPreprint
Atiq, A. (2026) · Research Square · doi:10.21203/rs.3.rs-11020069/v1
Executable but Unlearnable: Designing Code That Resists LLM-Based Learning
Mothukuri, V., & Parizi, R. M. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814896
Moraglio, A., & Tonda, A. P. (2026) · Genetic and evolutionary computation · doi:10.1007/978-981-95-6398-2_11
Balancing usefulness and naturalness: an LLM-based curation pipeline for code review comments
Sghaier, O. B., Weyssow, M., & Sahraoui, H. A. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10928-x
Alternative UX Extensions and Their Trade-offs for Code Completion in PharoPreprint
Elkolei, M., AbedelKader, O., & Ducasse, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.24253
Metadata
- Authors
- Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- The University of Texas at Dallas (US)
- Venue
- 2012 34th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2012-06
- Pages
- 837-847
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W3146720657
- Cited by
- 425
- References
- 39
- Citations by year
- 2026: 19 · 2025: 24 · 2024: 36 · 2023: 44 · 2022: 49 · 2021: 50 · 2020: 48 · 2019: 46 · 2018: 22 · 2017: 35 · 2016: 13 · 2015: 23 · 2014: 2 · 2013: 14
- FWCI
- 9.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support, natural language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.