Assessing the quality of GitHub copilot’s code generation
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
The introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large number of open-source projects, enabling it to utilize more extensive code in various programming languages than other code generation tools. Although the initial and informal assessments are promising, a systematic evaluation is needed to explore the limits and benefits of GitHub Copilot. The main objective of this study is to assess the quality of generated code provided by GitHub Copilot. We also aim to evaluate the impact of the quality and variety of input parameters fed to GitHub Copilot. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. Our results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 47 (28.7%) were correctly, while 84 (51.2%) were partially correctly, and 33 (20.1%) were incorrectly generated. Our empirical analysis shows that GitHub Copilot is a promising tool based on the results we obtained, however further and more comprehensive assessment is needed in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“[9] evaluated GitHub Copilot using 33 different LeetCode questions and four different programming languages (Python, Java, JavaScript, and C).”
MethodIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 1×
“[15] for their code generation and retrieval tool, the scope of the problems consists of basic Python, file, OS, web scraping, web server & client, data analysis & ML, and data visualization.”
MethodAI-Driven Development Is Here: Should You Worry?
Ernst, N. A., & Bavota, G. (2022) · IEEE Software · doi:10.1109/ms.2021.3133805 · Cited by 62
Research works printed but not linked: 6
- Shirley Anugrah Hayati Raphael Olivier Pravalika Avvaru Pengcheng Yin Anthony Tomasic and Graham Neubig. 2018. Retrieval-Based Neural Code Generation. https://doi.org/10.48550/ARXIV.1808.10025 · DOI
- 10.1007/s10664-021-09968-2 · DOI
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. https://doi.org/10.48550/ARXIV.2108.09293 · DOI
- 10.1145/3512290.3528700 · DOI
- 10.1609/aaai.v34i05.6430 · DOI · Paper page
- Maosheng Zhong Gen Liu Hongwei Li Jiangling Kuang Jinshan Zeng and Mingwen Wang. 2022. CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information. https://doi.org/10.48550/ARXIV.2202.07612 · arXiv · doi:10.48550/arxiv.2202.07612
Cited by
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
“Some current publications report that AI programming assistants are powerful enough to produce high-quality code suggestions for developers [55, 57].”
BackgroundA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
Significant Productivity Gains through Programming with Large Language Models
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3661145 · Cited by 64
“Using the Codex system, GitHub Copilot can directly integrate into coding editors like Visual Studio Code or IDEs [67].”
MethodAccelerating Software Development Using Generative AI: ChatGPT Case Study
Rajbhoj, A., Somase, A., Kulkarni, P., & Kulkarni, V. (2024) · Proceedings of the 17th Innovations in Software Engineering Conference · doi:10.1145/3641399.3641403 · Cited by 50
“Many researchers have successfully used Codex to generate accurate and efficient code snippets [15, 16].”
MethodUnveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38
“Yetistiren et al. [115] set out to evaluate the quality of code generated by GitHub Copilot.”
Background
Metadata
- Authors
- Yetistiren, B., Ozsoy, I., & Tuzun, E.
- Venue
- Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 62-71
- OpenAlex
- W4308627320
- Cited by
- 143
- References
- 5
- FWCI
- 26.53
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing the quality of GitHub copilot’s code generation. https://app.cochat.ai/papers/10.1145/3558489.3559072
- The paper
- Yetistiren, B., Ozsoy, I., & Tuzun, E. (2022). Assessing the quality of GitHub copilot’s code generation. Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering. https://doi.org/10.1145/3558489.3559072
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.