An empirical evaluation of GitHub copilot's code suggestions
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
GitHub and OpenAI recently launched Copilot, an "AI pair programmer" that utilizes the power of Natural Language Processing, Static Analysis, Code Synthesis, and Artificial Intelligence. Given a natural language description of the target functionality, Copilot can generate corresponding code in several programming languages. In this paper, we perform an empirical study to evaluate the correctness and understandability of Copilot's suggested code. We use 33 LeetCode questions to create queries for Copilot in four different programming languages. We evaluate the correctness of the corresponding 132 Copilot solutions by running LeetCode's provided tests, and evaluate understandability using SonarQube's cyclomatic complexity and cognitive complexity metrics. We find that Copilot's Java suggestions have the highest correctness score (57%) while JavaScript is the lowest (27%). Overall, Copilot's suggestions have low complexity with no notable differences between the programming languages. We also find some potential Copilot shortcomings, such as generating code that can be further simplified and code that relies on undefined helper methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Subsequently, the quality of the code generated by LLMs also cannot be guaranteed [23], [24], [25], [26].”
From No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT (2024) · cites this paper as [24]
In the citing paper
“Research has shown that such tools may produce insecure or suboptimal code, requiring users to review and revise outputs carefully to avoid compounding mistakes (Groothuijsen et al., 2024;Laux, 2024;Nguyen & Nadi, 2022).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Nguyen & Nadi, 2022
In the citing paper
“Nguyen et al. [37] evaluated the performance of Copilot on 33 programming tasks.”
From Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues (2024) · cites this paper as [37]
In the citing paper
“Moreover, Nguyen et al. [37] focus on analyzing the correctness of AI-generated code while Fan et al. [15] target to benchmark automated program repair tools on ixing mistakes.”
From Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues (2024) · cites this paper as [37]
In the citing paper
“However, these studies focus on CodeX and GitHub Copilot on a few programming tasks, leading to a lack of diversity.”
From Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues (2024) · cites this paper as [37]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
code2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962
Gu, X., Zhang, H., & Kim, S. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180167 · Cited by 547
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Recommendation Systems for Software Engineering
Robillard, M., Walker, R., & Zimmermann, T. (2010) · IEEE Software · doi:10.1109/ms.2009.161 · Cited by 388
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
When deep learning met code search
Cambronero, J., Li, H., Kim, S., Sen, K., & Chandra, S. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3340458 · Cited by 214
Cognitive complexity: an overview and evaluation
Campbell, G. A. (2018) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3194164.3194186 · Cited by 182
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150 · cited 1×
“There has been lots of research dedicated to improving developer productivity through code synthesis, code search, or other types of code recommender systems [2, 4, 20, 25].”
BackgroundCode Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., & King, I. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/578 · Cited by 112
Aroma: code recommendation via structural code search
Luan, S., Yang, D., Barnaby, C., Sen, K., & Chandra, S. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360578 · Cited by 104
Sobania, D., Briesch, M., & Rothlauf, F. (2022) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3512290.3528700 · Cited by 91
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
Yao, Z., Peddamail, J. R., & Sun, H. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313632 · Cited by 89
PeerLens: Peer-inspired Interactive Learning Path Planning in Online Question Pool
Xia, M., Sun, M., Wei, H., Chen, Q., Wang, Y., Shi, L., Qu, H., & Ma, X. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300864 · Cited by 49
A Syntax-Guided Edit Decoder for Neural Program RepairPreprint
Zhu, Q., Sun, Z., Xiao, Y.-A., Zhang, W., Yuan, K., Xiong, Y., & Zhang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08253 · Cited by 26
Readability and Understandability Scores for Snippet Assessment: an Exploratory StudyPreprint
Dantas, C. E. C., & Maia, M. D. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09181 · Cited by 2
An Empirical Evaluation of GitHub Copilot’s Code Suggestions
Nguyen, N., & Nadi, S. (2022) · Figshare · doi:10.6084/m9.figshare.18515141 · Cited by 1
Self-citation
Cited by
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Evaluations have often relied on small programming problems from programming courses [86], synthetically generated problem sets [87], and online judging platforms such as Leetcode [28], [64], [88].”
BackgroundThe promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Metadata
- Authors
- Nguyen, N., & Nadi, S.
- Affiliations
- University of Alberta (CA)
- ORCID
- Nadi, Sarah
- Venue
- Proceedings of the 19th International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2022-05-23
- Pages
- 1-5
- OpenAlex
- W4312438588
- Cited by
- 313
- References
- 14
- Citations by year
- 2026: 47 · 2025: 100 · 2024: 99 · 2023: 65 · 2022: 1
- FWCI
- 59.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- Funders
- Canada Research Chairs Program
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, AI-assisted programming, code generation, program correctness, code understandability, cyclomatic complexity, cognitive complexity, LeetCode, programming languages, JavaScript, Java
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.