CoChat
Sign Up
Cited by 50Software Engineering ResearchSoftware Engineering Techniques and PracticesSoftware Testing and Debugging Techniques

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

How Practitioners Expect Code Completion?

Open access
Wang, Chaozheng, Hu, Junhao, Gao, Cuiyun, Jin, Yu, Xie, Tao, Huang, Hailiang, Lei, Zhenyu, Deng, Yuetang·2023·Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
DOI 10.1145/3611643.3616280
Open this paper in CoChat — free accountOpen PDF

Abstract

Code completion has become a common practice for programmers during their daily programming activities. It automatically predicts the next tokens or statements that the programmers may use. Code completion aims to substantially save keystrokes and improve the programming efficiency for programmers. Although there exists substantial research on code completion, it is still unclear what practitioner expectations are on code completion and whether these expectations are met by the existing research. To address these questions, we perform a study by first interviewing 15 professionals and then surveying 599 practitioners from 18 IT companies about their expectations on code completion. We then compare the practitioner expectations with the existing research by conducting a literature review of papers on code completion published in major publication venues from 2012 to 2022. Based on the comparison, we highlight the directions desirable for researchers to invest efforts toward developing code completion techniques for meeting practitioner expectations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3611643.3616280

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Developers need finer-grained suggestions and trustworthy explanations, which are typically not provided in CCTs.

    “Other evidence about the misalignment between developers' mental models and CCTs regards the developers' need for finer-grained suggestions and trustworthy explanations (Sergeyuk et al., 2025; Wang et al., 2023), typically not provided in CCTs, as well as the different preferences about inline and sidebar layouts (Vaithilingam et al., 2023; M˘ ar˘ as , oiu et al., 2015).”

    From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Wang et al., 2023

    In RQ3, the paper reports demand for specific completion scenarios, including skeleton prediction and completion of the currently edited line; RQ4 says some of these receive little research attention. RQ3 also reports that practitioners want API definitions or documentation alongside suggestions, but it does not establish a need for trustworthy explanations or that code-completion tools typically lack them. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. 10.1109/SANER.2016.39 · DOI · Paper page
  2. Fabio Villamarin Arrebola and Plinio Thomaz Aquino Junior. 2017. On source code completion assistants and the need of a context-aware approach. In Human Interface and the Management of Information: Supporting Learning, Decision-Making and Collaboration: 19th International Conference, HCI International 2017, Vancouver, BC, Canada, July 9–14, Part II 19. 191–201.
  3. 10.1145/3540250.3558968 · DOI
  4. Pavol Bielik, Veselin Raychev, and Martin T. Vechev. 2016. PHOG: probabilistic model for code. In 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48). 2933–2942.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  6. 10.1145/1595696.1595728 · DOI
  7. 10.1109/MSR52588.2021.00024 · DOI
  8. 10.1016/j.infsof.2021.106690 · DOI
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). 4171–4186.
  10. Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, and Miltiadis Allamanis. 2021. Learning to complete code with sketches. In International Conference on Learning Representations.
  11. 10.1145/2491956.2462192 · DOI
  12. Vincent J. Hellendoorn and Premkumar T. Devanbu. 2017. Are deep neural networks the best choice for modeling source code? In 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. 763–773.
  13. Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar T. Devanbu. 2012. On the naturalness of software. In 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland. 837–847.
  14. Daqing Hou and David M. Pletcher. 2010. Towards a better code completion system by API grouping, filtering, and popularity-based ranking. In 2nd International Workshop on Recommendation Systems for Software Engineering, RSSE 2010, Cape Town, South Africa, May 4, 2010. 26–30.
  15. 10.1145/3510003.3510152 · DOI
  16. 10.1016/j.infsof.2020.106309 · DOI
  17. 10.1016/j.jss.2022.111553 · DOI
  18. 10.1145/3510003.3510172 · DOI · Paper page
  19. 10.1145/3526113.3545659 · DOI · Paper page
  20. 10.1109/TSE.2014.2318734 · DOI · Paper page
  21. 10.1109/ICSE43902.2021.00026 · DOI · Paper page
  22. Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. In 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016. 165–176.
  23. 10.1109/ICSME52107.2021.00024 · DOI
  24. 10.24963/ijcai.2018/578 · DOI
  25. 10.1145/3387904.3389261 · DOI
  26. Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. 2022. A unified multi-task learning model for AST-level and token-level code completion. Empir. Softw. Eng., 27, 91 (2022).
  27. 10.1145/3324884.3416591 · DOI · Paper page
  28. 10.1016/j.jss.2020.110547 · DOI
  29. 10.18653/v1/2022.acl-long.431 · DOI · Paper page
  30. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: a machine learning benchmark dataset for code understanding and generation. In Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  31. 10.1145/1065010.1065018 · DOI
  32. 10.1145/3544548.3580940 · DOI
  33. Anh Tuan Nguyen and Tien N. Nguyen. 2015. Graph-based statistical language model for code. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16-24, 2015, Volume 1. 858–868.
  34. 10.1145/3551349.3559544 · DOI
  35. 10.1109/ASE.2019.00072 · DOI
  36. Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2013. A statistical semantic language model for source code. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2013, Saint Petersburg, Russian Federation, August 18-26, 2013. 532–542.
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In 40th Annual Meeting of the Association for Computational Linguistics, ACL 2002, Philadelphia, PA, USA. 311–318.
  38. 10.1145/3527329 · DOI
  39. 10.1145/2254064.2254098 · DOI
  40. Sebastian Proksch, Johannes Lerch, and Mira Mezini. 2016. Intelligent code completion with Bayesian networks. In Software Engineering 2016, Fachtagung des GI-Fachbereichs Softwaretechnik, 23.-26. Februar 2016, Wien, Österreich (LNI, Vol. P-252). 25–26.
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
  42. Veselin Raychev, Pavol Bielik, and Martin T. Vechev. 2016. Probabilistic model for code with decision trees. In 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016, Amsterdam, The Netherlands, October 30 - November 4, 2016. 731–747.
  43. 10.1145/2594291.2594321 · DOI · Paper page
  44. 10.1016/j.jss.2016.12.036 · DOI
  45. 10.1016/j.jss.2016.06.063 · DOI
  46. Alex Sherstinsky. 2018. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. CoRR, abs/1808.03314 (2018). · arXiv
  47. Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. IntelliCode compose: code generation using transformer. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, Virtual Event, USA, November 8-13, 2020. 1433–1443.
  48. 10.1109/MSR52588.2021.00045 · DOI · Paper page
  49. 10.1145/3292500.3330699 · DOI · Paper page
  50. 10.1145/1321631.1321663 · DOI
  51. Zhaopeng Tu, Zhendong Su, and Premkumar T. Devanbu. 2014. On the localness of software. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE 2014, Hong Kong, China, November 16 - 22, 2014. 269–280.
  52. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. experience: evaluating the usability of code generation tools powered by large language models. In CHI 2022: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Extended Abstracts. 332:1–332:7.
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems NeurIPS 2017, December 4-9, Long Beach, CA, USA. 5998–6008.
  54. 10.1609/aaai.v35i16.17650 · DOI
  55. 10.1109/ICSE43902.2021.00025 · DOI
  56. 10.1109/ASE.2017.8115678 · DOI
  57. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022. 21–29.

Cited by

Sorted by citations

Metadata

Authors
Wang, C., Hu, J., Gao, C., Jin, Y., Xie, T., Huang, H., Lei, Z., & Deng, Y.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Peking University (CN)
  • Tencent, Guangzhou, USA
  • Tencent (China) (CN)
ORCID
Venue
Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2023-11-30
Pages
1294-1306
DOI
10.1145/3611643.3616280
OpenAlex
W4389159031
Cited by
50
References
4
Citations by year
2026: 16 · 2025: 18 · 2024: 12 · 2023: 4
FWCI
9.62
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62161146003)
  • Tencent Foundation (XPLORER PRIZE)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
code completion, practitioner perspectives, next-token prediction, programming efficiency, developer interviews, research directions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review