Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
AI programming has become a popular topic in recent years. Code suggestion, with code suggestion being a key capability of AI programming. Copilot, an “AI programmer” that provides code suggestions from natural language descriptions, has been launched by GitHub and OpenAI. By far, Copilot has been widely used by millions of developers. However, little work has systematically evaluated the correctness of Copilot's suggestions. We conducted an empirical study on all 2,033 LeetCode problems to assess Copilot's code generation across four mainstream languages: C, Java, JavaScript, and Python. We have found that: 1) 70.0% of problems received at least one correct suggestion, with language-specific rates of 29.7% (C), 57.7% (Java), 54.1% (JavaScript), and 41.0% (Python); 2) Correctness decreases as problem difficulty increases, with acceptance rates of 89.3% (Easy), 72.1% (Medium), and 43.4% (Hard); 3) Acceptance rates vary across problem domains from 49.5% to 90.1%, while Graph problems challenge C and Python most, and Prefix Sum and Heap challenge Java and JavaScript most; 4) For the incorrect suggestions, we further summarize 17 types of error reasons accounting for their incorrectness and analyzed possible causes for why these errors occur. We believe our study can provide valuable insights into Copilot's capabilities and limitations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,635
O’Regan, G. (2023) · Texts in Computer Science · doi:10.1007/978-3-031-26212-8_4 · Cited by 1,614
A Normalized Levenshtein Distance Metric
Yujian, L., & Bo, L. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1078 · Cited by 986 · cited 1×
“To proceed with our study, we seek to answer four RQs:”
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 541
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 74–81.
- Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of International Conference on Machine Learning. PMLR, 11328–11339.
Cited by
AI, Jobs, and the Automation Trap: Where Is HCI?
Constantinides, M., & Quercia, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729191 · Cited by 19
Barari, G., Ortega-Moody, J., Jenab, K., Ward, T., & Siebold, K. (2026) · Applied Sciences · doi:10.3390/app16104884 · Cited by 2
Can We Trust the AI Pair Programmer? Copilot for API Misuse Detection and Correction
Mondal, S., Roy, C. K., Wang, H., Arguello, J., & Mathan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00025 · Cited by 2
Malashin, I. P. (2025) · Computers · doi:10.3390/computers14120534 · Cited by 1
Toward Explaining Large Language Models in Software Engineering Tasks
Vitale, A., Nguyen, K.-N., Poshyvanyk, D., Oliveto, R., Scalabrino, S., & Mastropaolo, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3837084 · Cited by 2
Embracing large language models in education: insights and reflections from the Caribbean context
Ildephonce, I., Hunte, A. A., & Rolle-Greenidge, G. (2026) · Caribbean Educational Research Journal · doi:10.1108/cerj-04-2026-0009
Architectural Constraints Alignment in AI-Assisted, Platform-Based Service Development
Irion, J., Leugers, M., Hartwig, P., Kling, S., Annayev, T., Schwind, A., Borges, M. C., & Werner, S. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-28160-9_14
Grasping AI Reliance in Program Comprehension and Coding through the AIRELI Persona Taxonomy
Alakmeh, T., Anderson, N. B., Jackson, V., Pereira, G. V., Akırmak, Ü., Estey, A., Prikladnicki, R., van der Hoek, A., Storey, M., & Fritz, T. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794804 · Cited by 5
Pragmatic Use of Generative AI in an Introductory Programming Course (extended abstract)
Klein, T. O. V., & Zaidman, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805778
When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code
Pădurean, V.-A., Riegel, K., Gotovos, A., Mahapatra, J., Ghosh, A., Denny, P. C., Leinonen, J., Prather, J. E., & Singla, A. K. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811667 · Cited by 4
Wen, Y., Cui, Z., Liu, Y., Zhang, Z., Zhou, J., & Tang, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07487-1 · Cited by 1
Metadata
- Authors
- Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y.
- Affiliations
- Central China Normal University (CN)
- University of South Carolina (US)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-08-14
- Volume
- 34
- Issue
- 7
- Pages
- 1-32
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4406866173
- Cited by
- 11
- References
- 31
- Citations by year
- 2026: 7 · 2025: 4
- FWCI
- 11.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- GitHub Copilot, code completion, AI programming, problem difficulty, problem domain, error types, programming languages, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.