The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Abstract
AI programming has become a popular topic in recent years. Code suggestion, with code suggestion being a key capability of AI programming. Copilot, an “AI programmer” that provides code suggestions from natural language descriptions, has been launched by GitHub and OpenAI. By far, Copilot has been widely used by millions of developers. However, little work has systematically evaluated the correctness of Copilot's suggestions. We conducted an empirical study on all 2,033 LeetCode problems to assess Copilot's code generation across four mainstream languages: C, Java, JavaScript, and Python. We have found that: 1) 70.0% of problems received at least one correct suggestion, with language-specific rates of 29.7% (C), 57.7% (Java), 54.1% (JavaScript), and 41.0% (Python); 2) Correctness decreases as problem difficulty increases, with acceptance rates of 89.3% (Easy), 72.1% (Medium), and 43.4% (Hard); 3) Acceptance rates vary across problem domains from 49.5% to 90.1%, while Graph problems challenge C and Python most, and Prefix Sum and Heap challenge Java and JavaScript most; 4) For the incorrect suggestions, we further summarize 17 types of error reasons accounting for their incorrectness and analyzed possible causes for why these errors occur. We believe our study can provide valuable insights into Copilot's capabilities and limitations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
O’Regan, G. (2023) · Texts in Computer Science · doi:10.1007/978-3-031-26212-8_4 · Cited by 1,614
A Normalized Levenshtein Distance Metric
Yujian, L., & Bo, L. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1078 · Cited by 986
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 541
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00181 · Cited by 296
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292
Tu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191
Multi-task Learning based Pre-trained Language Model for Code Completion
Liu, F., Li, G., Zhao, Y., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416591 · Cited by 188
Self-citationBeyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.00971 · Cited by 188
Self-citationApplying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186
Aroma: code recommendation via structural code search
Luan, S., Yang, D., Barnaby, C., Sen, K., & Chandra, S. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360578 · Cited by 178
General Program Synthesis Benchmark Suite
Helmuth, T., & Spector, L. (2015) · Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation · doi:10.1145/2739480.2754769 · Cited by 172
Towards Better Chain-of-Thought Prompting Strategies: A SurveyPreprint
Yu, Z., He, L., Wu, Z., Dai, X., & Chen, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.04959 · Cited by 161
Sobania, D., Briesch, M., & Rothlauf, F. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528700 · Cited by 155
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150 · cited 1×
“Code suggestion [1, 30, 33, 35] is one of the most active topics in recent years.”
An Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 134
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
An Empirical Study on the Usage of BERT Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00024 · Cited by 126
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116
Improving code completion with program history
Robbes, R., & Lanza, M. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0064-x · Cited by 95
PSB2: the second program synthesis benchmark suite
Helmuth, T., & Kelly, P. (2021) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3449639.3459285 · Cited by 71
Maybe Deep Neural Networks are the Best Choice for Modeling Source CodePreprint
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05734 · Cited by 64
PeerLens: Peer-inspired Interactive Learning Path Planning in Online Question Pool
Xia, M., Sun, M., Wei, H., Chen, Q., Wang, Y., Shi, L., Qu, H., & Ma, X. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300864 · Cited by 59
Self-citationCrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793747 · Cited by 53
CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code
Eghbali, A., & Pradel, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556903 · Cited by 42
A unified multi-task learning model for AST-level and token-level code completion
Liu, F., Li, G., Wei, B., Xia, X., Fu, Z., & Jin, Z. G. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10140-7 · Cited by 37
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
van Dam, T., Izadi, M., & Arie Van Deursen, A. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00035 · Cited by 25
Code recommendation for exception handling
(2020) · doi:10.1145/3368089.3409690 · Cited by 22
On the naturalness of software
(2013) · doi:10.1145/2442754.2442763 · Cited by 11
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023) · arXiv · doi:10.48550/arxiv.2308.02828
Code Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., et al. (2017) · arXiv · doi:10.48550/arxiv.1711.09573
Self-citation
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 74–81.
- Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of International Conference on Machine Learning. PMLR, 11328–11339.
Cited by
AI, Jobs, and the Automation Trap: Where Is HCI?
Constantinides, M., & Quercia, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729191 · Cited by 19
Barari, G., Ortega-Moody, J., Jenab, K., Ward, T., & Siebold, K. (2026) · Applied Sciences · doi:10.3390/app16104884 · Cited by 2
Can We Trust the AI Pair Programmer? Copilot for API Misuse Detection and Correction
Mondal, S., Roy, C. K., Wang, H., Arguello, J., & Mathan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00025 · Cited by 2
Malashin, I. P. (2025) · Computers · doi:10.3390/computers14120534 · Cited by 1
Toward Explaining Large Language Models in Software Engineering Tasks
Vitale, A., Nguyen, K.-N., Poshyvanyk, D., Oliveto, R., Scalabrino, S., & Mastropaolo, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3837084 · Cited by 2
Metadata
- Authors
- Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y.
- Affiliations
- Central China Normal University (CN)
- University of South Carolina (US)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-08-14
- Volume
- 34
- Issue
- 7
- Pages
- 1-32
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4406866173
- Cited by
- 11
- References
- 31
- Citations by year
- 2026: 7 · 2025: 4
- FWCI
- 11.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- GitHub Copilot, code completion, AI programming, problem difficulty, problem domain, error types, programming languages, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.