CoChat

Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions

Mo, Ran, Wang, Dongyu, Zhan, Wenjing, Jiang, Yingjie, Wang, Yepeng, Zhao, Yuqi, Li, Zengyang, Ma, Yutao · 2025 · ACM Transactions on Software Engineering and Methodology

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

AI programming has become a popular topic in recent years. Code suggestion, with code suggestion being a key capability of AI programming. Copilot, an “AI programmer” that provides code suggestions from natural language descriptions, has been launched by GitHub and OpenAI. By far, Copilot has been widely used by millions of developers. However, little work has systematically evaluated the correctness of Copilot's suggestions. We conducted an empirical study on all 2,033 LeetCode problems to assess Copilot's code generation across four mainstream languages: C, Java, JavaScript, and Python. We have found that: 1) 70.0% of problems received at least one correct suggestion, with language-specific rates of 29.7% (C), 57.7% (Java), 54.1% (JavaScript), and 41.0% (Python); 2) Correctness decreases as problem difficulty increases, with acceptance rates of 89.3% (Easy), 72.1% (Medium), and 43.4% (Hard); 3) Acceptance rates vary across problem domains from 49.5% to 90.1%, while Graph problems challenge C and Python most, and Prefix Sum and Heap challenge Java and JavaScript most; 4) For the incorrect suggestions, we further summarize 17 types of error reasons accounting for their incorrectness and analyzed possible causes for why these errors occur. We believe our study can provide valuable insights into Copilot's capabilities and limitations.

Overview

Integrity signals

Registry

Crossref · 10.1145/3715108

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 40 references

Research works printed but not linked: 2

  1. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 74–81.
  2. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of International Conference on Machine Learning. PMLR, 11328–11339.

Cited by

See all 11

Metadata

Authors
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y.
Affiliations
  • Central China Normal University (CN)
  • University of South Carolina (US)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2025-08-14
Volume
34
Issue
7
Pages
1-32
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4406866173
Cited by
11
References
31
Citations by year
2026: 7 · 2025: 4
FWCI
11.53
Type
journal article
Language
en
Open access
closed
Locations
ACM Transactions on Software Engineering and Methodology (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
GitHub Copilot, code completion, AI programming, problem difficulty, problem domain, error types, programming languages, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.