On the naturalness of software
The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Abstract
Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural , in the sense that it is created by humans at work, with all the attendant constraints and limitations---and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether (a) code can be usefully modeled by statistical language models and (b) such models can be leveraged to support software engineers. Using the widely adopted n -gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very regular, and, in fact, even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's completion capability. We conclude the paper by laying out a vision for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
On The Language Instinct: (412952005-009)
Pinker, S., & American Psychological Association (1994) · PsycEXTRA Dataset · doi:10.1037/e412952005-009 · Cited by 5,646
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Knuth, D. E. (1984) · The Computer Journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,467
Statistical machine translation
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6293 · Cited by 1,309
Recovering traceability links between code and documentation
Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., & Merlo, E. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1041053 · Cited by 967
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Mining Version Histories to Guide Software Changes
Zimmermann, T., Member, S., Weißgerber, P., Diehl, S., Zeller, A., & Ieee Computer Society (2004) · Cited by 494
Building a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 489
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
DynaMine: finding common error patterns by mining software revision histories
Livshits, B., & Zimmermann, T. (2005) · Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1081706.1081754 · Cited by 330
Real-time specification patterns
Konrad, S., & Cheng, B. H. C. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553580 · Cited by 307
Statistical language model adaptation: review and perspectives
Bellegarda, J. R. (2004) · Speech Communication · doi:10.1016/j.specom.2003.08.002 · Cited by 297
Predicting Program Properties from "Big Code"
Raychev, V., Vechev, M., & Krause, A. (2015) · Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/2676726.2677009 · Cited by 296
Whats in a Name? A Study of Identifiers
Lawrie, D., Morrell, C., Feild, H., & Binkley, D. (2006) · 14th IEEE International Conference on Program Comprehension (ICPC'06) · doi:10.1109/icpc.2006.51 · Cited by 248
Tu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · cited 3×
“However, we note that these are very early efforts in this area, and new software language models 35, 49 and estimation techniques have improved on the results presented below.”
BackgroundSelf-citationSourcerer: mining and searching internet-scale software repositories
Linstead, E., Bajracharya, S., Ngo, T., Rigor, P., Lopes, C., & Baldi, P. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0118-x · Cited by 237
Automatically documenting program changes
Buse, R. P. L., & Weimer, W. R. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859005 · Cited by 219
A study of the uniqueness of source code
Gabel, M., & Su, Z. (2010) · Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1882291.1882315 · Cited by 215
Self-citationA statistical semantic language model for source code
Nguyen, T. T., Nguyen, A. T., Nguyen, H. A., & Nguyen, T. N. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491458 · Cited by 208
Using natural language program analysis to locate and understand action-oriented concerns
Shepherd, D., Fry, Z. P., Hill, E., Pollock, L., & Vijay-Shanker, K. (2007) · Proceedings of the 6th international conference on Aspect-oriented software development · doi:10.1145/1218563.1218587 · Cited by 205
Toward deep learning software repositories
White, M., Vendome, C., & Poshyvanyk, D. (2015) · Cited by 192
Javert: fully automatic mining of general temporal properties from dynamic traces
Gabel, M., & Su, Z. (2008) · Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering · doi:10.1145/1453101.1453150 · Cited by 190
Self-citationLexical statistical machine translation for language migration
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494584 · Cited by 189
Automatically detecting and describing high level actions within methods
Sridhara, G., Pollock, L., & Vijay-Shanker, K. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985808 · Cited by 182
Natural Language Processing: A Historical Review
Jones, K. S. (1994) · Springer eBooks · doi:10.1007/978-0-585-35958-8_1 · Cited by 181
Høst, E. W., & Østvold, B. M. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03013-0_14 · Cited by 170
A natural language approach for Requirements Engineering
Rolland, C., & Proix, C. (1992) · Notes on Numerical Fluid Mechanics and Multidisciplinary Design · doi:10.1007/bfb0035136 · Cited by 167
(2014) · doi:10.1145/2635868.2635901 · Cited by 159
Kim, S., Pan, K., & Whitehead, E. J. (2006) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1181775.1181781 · Cited by 152
Graph-based statistical language model for code
Nguyen, A. T., & Nguyen, T. N. (2016) · Cited by 142
Data Mining for Software Engineering
Xie, T., Thummalapenta, S., Lo, D., & Liu, C. (2009) · Computer · doi:10.1109/mc.2009.256 · Cited by 141
Phrase-Based Statistical Translation of Programming Languages
Karaivanov, S., Raychev, V., & Vechev, M. (2014) · Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software · doi:10.1145/2661136.2661148 · Cited by 114
Graph-Based Statistical Language Model for Code
Nguyen, A. T., & Nguyen, T. N. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.336 · Cited by 114
Statistical learning approach for mining API usage mappings for code migration
Nguyen, A. T., Nguyen, H. A., Nguyen, T. T., & Nguyen, T. N. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2643010 · Cited by 103
DynaMine: finding common error patterns by mining software revision histories
Livshits, B., & Zimmermann, T. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1095430.1081754 · Cited by 96
Improving code completion with program history
Robbes, R., & Lanza, M. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0064-x · Cited by 95
Syntax errors just aren't natural: improving error reporting with language models
Campbell, J. C., Hindle, A., & Amaral, J. N. (2014) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2597073.2597102 · Cited by 83
Self-citationGenerating natural language summaries for crosscutting source code concerns
Rastkar, S., Murphy, G. C., & Bradley, A. W. J. (2011) · 2011 27th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080777 · Cited by 80
An evaluation of the strategies of sorting, filtering, and grouping API methods for Code Completion
Hou, D., & Pletcher, D. M. (2011) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080790 · Cited by 72
Improving identifier informativeness using part of speech information
Binkley, D., Hearn, M., & Lawrie, D. (2011) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/1985441.1985471 · Cited by 72
Code Completion from Abbreviated Input
Han, S., Wallace, D. R., & Miller, R. C. (2009) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2009.64 · Cited by 71
Migrating code with statistical machine translation
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591072 · Cited by 67
The Language Instinct. The New Science of Language and Mind. London (Penguin) 1994.
Pinker, S. A. (1994) · Cited by 52
Using language clues to discover crosscutting concerns
Shepherd, D. C., Pollock, L., & Tourwé, T. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083129 · Cited by 43
Programming by voice, VocalProgramming
Arnold, S. C., Mark, L., & Goldthwaite, J. (2000) · Proceedings of the fourth international ACM conference on Assistive technologies · doi:10.1145/354324.354362 · Cited by 39
Spoken Language Support for Software Development
Begel, A. (2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2004.49 · Cited by 36
The Java Programmer’s Phrase Book
Høst, E. W., & Østvold, B. M. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00434-6_20 · Cited by 35
CACHECA: A Cache Language Model Based Code Suggestion Tool
Franks, C., Tu, Z., Dévanbu, P., & Hellendoorn, V. J. (2015) · Cited by 34
Bruch, M., Bodden, E., Monperrus, M., & Mezini, M. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882374 · Cited by 30
Code template inference using language models
Jacob, F., & Tairas, R. (2010) · Proceedings of the 48th Annual Southeast Regional Conference · doi:10.1145/1900008.1900143 · Cited by 24
Text to software: developing tools to close the gaps in software engineering
Tichy, W. F., & Köerner, S. J. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882439 · Cited by 16
A voice-activated syntax-directed editor for manually disabled programmers
Hubbell, T. J., Langan, D. D., & Hain, T. F. (2006) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1168987.1169022 · Cited by 13
Is Voice Recognition the Solution to Keyboard-Based RSI?
Mills, S., Saadat, S., & Whiting, D. (2006) · World Automation Congress (WAC) · doi:10.1109/wac.2006.375941 · Cited by 5
Research works printed but not linked: 6
- 10.5555/2819009.2819143 · doi:10.5555/2819009.2819143
- 10.5555/1734086 · doi:10.5555/1734086
- 10.1145/1062455.1062526 · doi:10.1145/1062455.1062526 · Paper page
- 10.5555/311445 · doi:10.5555/311445
- 10.5555/2820518.2820559 · doi:10.5555/2820518.2820559
- 10.5555/998675.999460 · doi:10.5555/998675.999460
Cited by
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Automating code review activities by large-scale pre-training
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549081 · Cited by 196
VulRepair: a T5-based automated software vulnerability repair
Fu, M., Tantithamthavorn, C., Le, T., Nguyen, V., & Phung, D. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549098 · Cited by 207
Neuro-Symbolic Program SynthesisPreprint
Parisotto, E., Mohamed, A., Singh, R., Li, L. C., Zhou, D., & Kohli, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01855 · Cited by 303
ChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
Metadata
- Authors
- Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P.
- Affiliations
- University of Alberta (CA)
- University College London (GB)
- University of California, Davis (US)
- UC Davis, CA
- Venue
- Communications of the ACM
- Published
- 2016-04-26
- Volume
- 59
- Issue
- 5
- Pages
- 122-131
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2344444819
- Cited by
- 307
- References
- 56
- Citations by year
- 2026: 34 · 2025: 58 · 2024: 48 · 2023: 45 · 2022: 66 · 2021: 26 · 2020: 16 · 2019: 5 · 2018: 6 · 2017: 2 · 2016: 1
- FWCI
- 10.42
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/J017515/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- statistical language models, n-gram language models, code completion, Java, eclipses, software engineering support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.