On the localness of software
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
The n-gram language model, which has its roots in statistical natural language processing, has been shown to successfully capture the repetitive and predictable regularities (“naturalness") of source code, and help with tasks such as code suggestion, porting, and designing assistive coding devices. However, we show in this paper that this natural-language-based model fails to exploit a special property of source code: localness. We find that human-written programs are localized: they have useful local regularities that can be captured and exploited. We introduce a novel cache language model that consists of both an n-gram and an added “cache" component to exploit localness. We show empirically that the additional cache component greatly improves the n-gram approach by capturing the localness of software, as measured by both cross-entropy and suggestion accuracy. Our model’s suggestion accuracy is actually comparable to a state-of-the-art, semantically augmented language model; but it is simpler and easier to implement. Our cache language model requires nothing beyond lexicalization, and thus is applicable to all programming languages.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Vocabularies of a project are highly local, and functions from the same projects are likely to share same set of identifiers [10,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [22]
In the citing paper
“It's been well-known right from the outset that language modeling for code has to deal with project-specific phenomena [10,11,22].”
From Few-shot training LLMs for project-specific code-summarization (2022) · cites this paper as [22]
Citations
Cites
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
A statistical approach to machine translation
Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J., Mercer, R. L., & Roossin, P. S. (1990) · doi:10.5555/92858.92860 · Cited by 1,655
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Katz, S. (1987) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1987.1165125 · Cited by 1,637
A Maximum Likelihood Approach to Continuous Speech Recognition
Bahl, L. R., Jelinek, F., & Mercer, R. L. (1983) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1983.4767370 · Cited by 1,212
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 541
Self-citationTowards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · cited 1×
“on GitHub and BitBucket), software engineering researchers have turned to empirical methods to study the process of developing and maintaining software [11, 27, 38, 40, 42–45, 47, 49].”
MethodSemantic clustering: Identifying topics in source code
Kuhn, A., Ducasse, S., & Gîrba, T. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2006.10.017 · Cited by 496
A cache-based natural language model for speech recognition
Kuhn, R., & De Mori, R. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.56193 · Cited by 495
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
Dixon, W. J., & Mood, A. M. (1946) · Journal of the American Statistical Association · doi:10.1080/01621459.1946.10501898 · Cited by 422
MAPO: Mining and Recommending API Usage Patterns
Zhong, H., Xie, T., Zhang, L., Pei, J., & Mei, H. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03013-0_15 · Cited by 410
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065018 · Cited by 388
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378
Learning natural coding conventions
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635883 · Cited by 376
The Modular Structure of Complex Systems
Parnas, D. L., Clements, P. C., & Weiss, D. M. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232209 · Cited by 366
Mining source code repositories at massive scale using language modeling
Allamanis, M., & Sutton, C. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624029 · Cited by 313
Real-time specification patterns
Konrad, S., & Cheng, B. H. C. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553580 · Cited by 307
Design of a linguistic statistical decoder for the recognition of continuous speech
Jelinek, F., Bahl, L., & Mercer, R. (1975) · IEEE Transactions on Information Theory · doi:10.1109/tit.1975.1055384 · Cited by 271
A spelling correction program based on a noisy channel model
Kernighan, M. D., Church, K. W., & Gale, W. A. (1990) · Proceedings of the 13th conference on Computational linguistics - · doi:10.3115/997939.997975 · Cited by 262
Jungloid mining: helping to navigate the API jungle
Mandelin, D., Xu, L., Bodík, R., & Kimelman, D. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065018 · Cited by 261
Automatically documenting program changes
Buse, R. P. L., & Weimer, W. R. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859005 · Cited by 219
A study of the uniqueness of source code
Gabel, M., & Su, Z. (2010) · Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1882291.1882315 · Cited by 215
Self-citationA statistical semantic language model for source code
Nguyen, T. T., Nguyen, A. T., Nguyen, H. A., & Nguyen, T. N. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491458 · Cited by 208
Using natural language program analysis to locate and understand action-oriented concerns
Shepherd, D., Fry, Z. P., Hill, E., Pollock, L., & Vijay-Shanker, K. (2007) · Proceedings of the 6th international conference on Aspect-oriented software development · doi:10.1145/1218563.1218587 · Cited by 205
Automatically detecting and describing high level actions within methods
Sridhara, G., Pollock, L., & Vijay-Shanker, K. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985808 · Cited by 182
Mining source code to automatically split identifiers for software analysis
Enslen, E., Hill, E., Pollock, L., & Vijay-Shanker, K. (2009) · 2009 6th IEEE International Working Conference on Mining Software Repositories · doi:10.1109/msr.2009.5069482 · Cited by 170
A natural language approach for Requirements Engineering
Rolland, C., & Proix, C. (1992) · Notes on Numerical Fluid Mechanics and Multidisciplinary Design · doi:10.1007/bfb0035136 · Cited by 167
Approximate Structural Context Matching: An Approach to Recommend Relevant Examples
Holmes, R., Walker, R. J., & Murphy, G. C. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.117 · Cited by 158
Lexical statistical machine translation for language migration
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2494584 · Cited by 136
Nomen est omen: analyzing the language of function identifiers
Caprile, B., & Tonella, P. (2003) · Proceedings - Working Conference on Reverse Engineering · doi:10.1109/wcre.1999.806952 · Cited by 133
Automatic parameter recommendation for practical API usage
Zhang, C., Yang, J., Zhang, Y., Fan, J., Zhang, X., Zhao, J., & Ou, P. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227136 · Cited by 105
Cognitive Perspectives on the Role of Naming in Computer Programs
Liblit, B., Begel, A., & Sweetser, E. E. (2006) · Cited by 104
Graph-based pattern-oriented, context-sensitive source code completion
Nguyen, A. T., Nguyen, T. T., Nguyen, H. A., Tamrawi, A., Nguyen, H. V., Al-Kofahi, J. M., & Nguyen, T. N. (2012) · doi:10.5555/2337223.2337232 · Cited by 102
Natural Language Models for Predicting Programming Comments
Movshovitz‐Attias, D., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6475913 · Cited by 100
Improving code completion with program history
Robbes, R., & Lanza, M. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0064-x · Cited by 95
Evaluating high accuracy retrieval techniques
Shah, C. S., & Croft, W. B. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1008996 · Cited by 82
Syntax errors just aren't natural: improving error reporting with language models
Campbell, J. C., Hindle, A., & Amaral, J. N. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597102 · Cited by 81
Generating natural language summaries for crosscutting source code concerns
Rastkar, S., Murphy, G. C., & Bradley, A. W. J. (2011) · 2011 27th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080777 · Cited by 80
Detecting API documentation errors
Zhong, H., & Su, Z. (2013) · Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications · doi:10.1145/2509136.2509523 · Cited by 80
Self-citationAn evaluation of the strategies of sorting, filtering, and grouping API methods for Code Completion
Hou, D., & Pletcher, D. M. (2011) · 2011 27th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080790 · Cited by 72
The maintenance problem of application software: An empirical analysis
Alkhatib, G. I. (1992) · Journal of Software Maintenance Research and Practice · doi:10.1002/smr.4360040203 · Cited by 64
Quantifying identifier quality: an analysis of trends
Lawrie, D., Feild, H. A., & Binkley, D. (2006) · Empirical Software Engineering · doi:10.1007/s10664-006-9032-2 · Cited by 61
TIDIER: an identifier splitting approach using speech recognition techniques
Guerrouj, L., Penta, M. D., Antoniol, G., & Guéhéneuc, Y. (2011) · Journal of Software Evolution and Process · doi:10.1002/smr.539 · Cited by 50
The modular structure of complex systems
Parnas, D. L., Clements, P., & Weiss, D. M. (1984) · doi:10.5555/800054.801999 · Cited by 46
Using language clues to discover crosscutting concerns
Shepherd, D., Pollock, L., & Tourwé, T. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083129 · Cited by 43
Programming by voice, VocalProgramming
Arnold, S. C., Mark, L., & Goldthwaite, J. (2000) · Proceedings of the fourth international ACM conference on Assistive technologies · doi:10.1145/354324.354362 · Cited by 39
Code template inference using language models
Jacob, F., & Tairas, R. (2010) · Proceedings of the 48th Annual Southeast Regional Conference · doi:10.1145/1900008.1900143 · Cited by 24
Inferring specifications for resources from natural language API documentation
Zhong, H., Zhang, L., Xie, T., & Mei, H. (2011) · Automated Software Engineering · doi:10.1007/s10515-011-0082-3 · Cited by 24
Automated Identifier Completion and Replacement
Abebe, S. L., & Tonella, P. (2013) · Proceedings of the ... European Conference on Software Maintenance and Reengineering/Proceedings of the European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2013.35 · Cited by 23
Combining Statistical and Syntactic Methods in Recognizing Handwritten Sentences
Srihari, R. K., & Baltus, C. M. (1992) · Cited by 20
Aalst, V. D. W. M. P., Aldred, L., Dumas, M., ter Hofstede, A. H. M., Persson, A., & Stirna, J. (2004) · Lecture notes in computer science · doi:10.1007/b98058 · Cited by 18
Text to software: developing tools to close the gaps in software engineering
Tichy, W. F., & Köerner, S. J. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882439 · Cited by 16
Research works printed but not linked: 4
- 10.5555/2487085.2487127 · doi:10.5555/2487085.2487127
- M. Broy , F. Deißenböck , and M. Pizka . A holistic approach to software quality at work. In 3WCSQ , 2005 . M. Broy, F. Deißenböck, and M. Pizka. A holistic approach to software quality at work. In 3WCSQ, 2005.
- K. Knight . Bayesian inference with tears . Tutorial Workbook , 2009 . K. Knight. Bayesian inference with tears. Tutorial Workbook, 2009.
- 10.1145/1062455.1062526 · doi:10.1145/1062455.1062526 · Paper page
Cited by
Automatically learning semantic features for defect prediction
Wang, S., Liu, T., & Tan, L. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884804 · Cited by 689
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558
“They are also limited by the amount of context they consider [75], yet the novel approach of casting applications in terms of “naturalness” in SE contexts has spawned many applications of this technology to new problems.”
BackgroundGu, X., Zhang, H., Zhang, D., & Kim, S. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950334 · Cited by 485
“Statistical language models have been adapted to many software engineering tasks [26] such as learning natural code conventions [5], code suggestion [44], and code completion [37].”
MethodA Convolutional Attention Network for Extreme Summarization of Source Code
Allamanis, M., Peng, H. M., & Sutton, C. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03001 · Cited by 126
“Research has mostly focused on tokenlevel (Nguyen et al., 2013; Tu et al., 2014) and syntax-level (Maddison & Tarlow, 2014) language models of code and translation between programming languages (Karaivanov”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
“However, we note that these are very early efforts in this area, and new software language models 35, 49 and estimation techniques have improved on the results presented below.”
BackgroundSelf-citation
Metadata
- Authors
- Tu, Z., Su, Z., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- University of California at Davis, USA
- Venue
- Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2014-11-11
- Pages
- 269-280
- OpenAlex
- W2165747537
- Cited by
- 238
- References
- 52
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 16 · 2023: 21 · 2022: 16 · 2021: 27 · 2020: 33 · 2019: 29 · 2018: 26 · 2017: 9 · 2016: 23 · 2015: 20
- FWCI
- 14.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- n-gram language models, source code, localness, code completion, lexicalization, cross-entropy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.