The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Learning from examples to improve code completion systems
Open accessAbstract
The suggestions made by current IDE's code completion features are based exclusively on static type system of the programming language. As a result, often proposals are made which are irrelevant for a particular working context. Also, these suggestions are ordered alphabetically rather than by their relevance in a particular context. In this paper, we present intelligent code completion systems that learn from existing code repositories. We have implemented three such systems, each using the information contained in repositories in a different way. We perform a large-scale quantitative evaluation of these systems, integrate the best performing one into Eclipse, and evaluate the latter also by a user study. Our experiments give evidence that intelligent code completion systems which learn from examples significantly outperform mainstream code completion systems in terms of the relevance of their suggestions and thus have the potential to enhance developers' productivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mining association rules between sets of items in large databases
Agrawal, R., Imieliński, T., & Swami, A. (1993) · Proceedings of the 1993 ACM SIGMOD international conference on Management of data · doi:10.1145/170035.170072 · Cited by 14,907
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413
How Program History Can Improve Code Completion
Robbes, R., & Lanza, M. (2008) · 2008 23rd IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2008.42 · Cited by 155
Research works printed but not linked: 19
- Overview of the Third Text REtrieval Conference (TREC-3) , Gaithersburg, MD, USA , 1990 . NIST. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD, USA, 1990. NIST.
- 10.5555/646152.758674 · doi:10.5555/646152.758674
- C. M. Bishop . Pattern Recognition and Machine Learning . Springer , 2006 . C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
- 10.1145/1188835.1188847 · DOI
- 10.1145/1454247.1454254 · DOI
- 10.1145/351936.351951 · doi:10.1145/351936.351951
- 10.1109/TIT.1967.1053964 · DOI · Paper page
- 10.1145/24634.24636 · DOI
- D. Gangopadhyay and S. Mitra . Design by framework completion . Automated Software Eng. , 3 ( 3/4 ): 219 -- 237 , 1996 . D. Gangopadhyay and S. Mitra. Design by framework completion. Automated Software Eng., 3(3/4):219--237, 1996.
- 10.5555/1025115.1025222 · doi:10.5555/1025115.1025222
- 10.1145/1062455.1062491 · DOI · Paper page
- 10.1145/141936.141943 · DOI
- 10.1145/1065010.1065018 · DOI · Paper page
- 10.1145/337180.337200 · doi:10.1145/337180.337200
- 10.1145/1167473.1167508 · DOI
- 10.1109/TSE.2006.94 · DOI
- 10.1145/1321631.1321663 · DOI · Paper page
- C. J. van Rijsbergen . Information retrieval . Butterworths , London , 1979 . C. J. van Rijsbergen. Information retrieval. Butterworths, London, 1979.
- 10.1145/355045.355053 · DOI
Cited by
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
“A large number of intelligent code completion systems for both statically and dynamically typed languages have been proposed [6, 11, 41, 42].”
BackgroundMethodOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
On the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
“Code completion models—autocomplete tools used by programmers to suggest subsequent source code tokens, such as the names of API calls—are commonly used to reduce the effort of software development (Robbes & Lanza, 2008; Bruch et al., 2009; Nguyen & Nguyen, 2015; Proksch et al., 2015; Franks et al., 2015).”
Tu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
Pythia: AI-assisted Code Completion System
Svyatkovskiy, A., Zhao, Y., Fu, S., & Sundaresan, N. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330699 · Cited by 169
How Secure is Code Generated by ChatGPT?
Khoury, R., Avila, A. R., Brunelle, J., & Camara, B. M. (2023) · 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc53992.2023.10394237 · Cited by 141
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Improving code completion with program history
Robbes, R., & Lanza, M. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0064-x · Cited by 95
Fast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150
CodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86
When Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76
NLP2Code: Code Snippet Content Assist via Natural Language Tasks
Campbell, B. A., & Treude, C. (2017) · 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2017.56 · Cited by 65
Significant Productivity Gains through Programming with Large Language Models
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661145 · Cited by 64
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
CSCC: Simple, Efficient, Context Sensitive Code Completion
Asaduzzaman, M., Roy, C. K., Schneider, K. A., & Hou, D. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.29 · Cited by 55
Ulfsnes, R., Moe, N. B., Stray, V., & Skarpen, M. (2024) · Generative AI for Effective Software Development · doi:10.1007/978-3-031-55642-5_10 · Cited by 62
How Practitioners Expect Code Completion?
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616280 · Cited by 50
A Toolkit for Generating Code Knowledge Graphs
Abdelaziz, I., Dolby, J., McCusker, J., & Srinivas, K. (2021) · Proceedings of the 11th Knowledge Capture Conference · doi:10.1145/3460210.3493578 · Cited by 46
Razzaq, A., Buckley, J., Lai, Q., Yu, T., & Botterweck, G. (2025) · ACM Computing Surveys · doi:10.1145/3687299 · Cited by 36
CACHECA: A Cache Language Model Based Code Suggestion Tool
Franks, C., Tu, Z., Devanbu, P., & Hellendoorn, V. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.228 · Cited by 30
Learning Autocompletion from Real-World Datasets
Aye, G. A., Kim, S., & Li, H. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00022 · Cited by 29
Evaluating the evaluations of code recommender systems: a reality check
Proksch, S., Amann, S., Nadi, S., & Mezini, M. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970330 · Cited by 26
Self-citationLong-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Clement, C., Lu, S., Liu, X., Tufano, M., Drain, D., Duan, N., Sundaresan, N., & Svyatkovskiy, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.387 · Cited by 12
Improving Code Autocompletion with Transfer Learning
Zhou, W., Kim, S., Murali, V., & Aye, G. A. (2021) · arXiv · doi:10.48550/arxiv.2105.05991
“Significant attention has been given to improving suggestion prediction through machine learning [3]–[6] by feeding code to models as a sequence of tokens or even AST nodes [7].”
Background
Metadata
- Authors
- Bruch, M., Monperrus, M., & Mezini, M.
- Venue
- Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
- Publisher
- ACM
- Published
- 2009-08-24
- Pages
- 213-222
- OpenAlex
- W1970607969
- Cited by
- 378
- References
- 5
- FWCI
- 10.46
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
