CoChat
Sign Up
Cited by 30Software Engineering ResearchSoftware Engineering Techniques and PracticesOpen Source Software Innovations

Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack Overflow

Open access
Zhou, Xiyu, Liang, Peng, Zhang, Beiqi, Li, Zengyang, Ahmad, Aakash, Shahin, Mojtaba, Waseem, Muhammad·2025·Journal of Systems and Software
DOI 10.1016/j.jss.2024.112204
Open this paper in CoChat — free accountOpen PDF

Integrity signals

Registry

Crossref · 10.1016/j.jss.2024.112204

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Compatibility Issue had fewer identified causes.

    “Fewer causes have been identified for Compatibility Issue.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The study found no main causes for User Experience Issue.

    “From the results of Table 3, we did not identify the main causes leading to User Experience Issue.”
    Quote in the paperNot a substantial claim
  • Findingown data

    CIE and ICS cause NONSENSICAL SUGGESTION.

    “CIE and UF are the causes leading to LOW QUALITY SUGGESTION, while CIE and ICS are the causes for NONSENSICAL SUGGESTION.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    A frequent NCE scenario involves users reaching the Copilot server via an HTTP proxy, which can cause SSL interception.

    “A common situation of NCE is that users connect to the Copilot server through an HTTP proxy, which may lead to the intercept of Secure Socket Layer (SSL).”
    Quote in the paperNot a substantial claim
  • Findingown data

    NCE is the primary cause of ACCESSING FAILURE.

    “while ACCESSING FAILURE is primarily attributed to NCE;”
    Quote in the paperNot a substantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 13
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingnone given

    The other eight cause types are less frequent but still offer insights into particular Copilot problems.

    “The remaining eight types of causes are less common, but can still provide insights into specific problems related to Copilot.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The study identified 497 solutions, addressing 36.7% of all problems, and grouped them into 11 types.

    “we identified a total of 497 solutions, which were used to address 36.7% of all problems, and categorized into 11 types as shown in Table 4.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some users expressed concerns about potential copyright and policy issues arising from how Copilot operates.

    “The way in which Copilot operates raises concerns regarding potential copyright and policy issues, as expressed by some users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    UU is identified as the direct cause for many users' AUTHENTICATION FAILURE and FUNCTIONALITY FAILURE with Copilot.

    “For instance, UU is identified as the direct cause for many users experiencing AUTHENTICATION FAILURE and FUNCTIONAL-ITY FAILURE when using Copilot.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    About 23.1% of Operation Issues stem from Copilot server errors (CIE), and the remainder come from the operating environment.

    “When Copilot users encounter Operation Issues, nearly one quarter (23.1%) of these problems are caused by errors originating from Copilot server (i.e.CIE), while the rest of such problems are induced by the environment on which Copilot operates.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For self-solved problems, common effective solutions were Modify Configuration/Setting (22.1%), Use Suitable Version (17.1%), and Reinstall/Restart/Reauthorize Copilot (12.1%).

    “When users tried to solve problems themselves, Modify Configuration/Setting (22.1%),Use Suitable Version (17.1%), and Reinstall/Restart/Reauthorize Copilot (12.1%) were commonly used as effective solutions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Some FUNCTION REQUEST items asked for project-wide suggestions, code explanation, and chat functionality, with chat already available in GitHub Copilot Chat.

    “More innovative demands involve the need for Copilot to provide suggestions according to the whole project, as well as features like code explanation and chat functionality which have been provided in GitHub Copilot Chat (Zhao, 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhao, 2023

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The study identified 391 causes, which came from 28.9% of all Copilot usage problems, and grouped them into 16 types.

    “we identified a total of 391 causes, which were collected from 28.9% of all problems related to Copilot usage, and categorized into 16 types as presented in”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Common FUNCTION REQUEST items included accepting Copilot suggestions word by word and specifying file types or code development scopes where Copilot should operate automatically.

    “Common requests include the ability to accept Copilot's suggestions word by word and to specify where Copilot should automatically operate in terms of file types or code development scopes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The three most frequent Operation Issue types were FUNCTIONALITY FAILURE (226), AUTHENTICATION FAILURE (198), and STARTUP ISSUE (193).

    “FUNCTIONALITY FAILURE (226), AUTHENTICATION FAIL-URE (198), and STARTUP ISSUE (193), are the top three types of such problems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the 24 POOR FUNCTIONALITY EXPERIENCE items, the most frequent issues were inline suggestions that disrupted coding (5) and difficulty accepting parts of suggested code (2).

    “Among the POOR FUNCTIONALITY EXPERIENCE (24), the most commonly reported problems involve Copilot's inline suggestions that cause disruptions to the coding process of users (5) and the inconvenience of not being able to accept certain portions of the suggested code (2).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingnone given

    When using Copilot, verifying code suggestions requires additional time.

    “For researchers, we found that additional time is required for code suggestion verification when utilizing Copilot”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • May combine two claims.

    Findingnone given

    Users request more customization options to tailor Copilot's behavior and greater control over the code content it generates.

    “Additionally, users are asking for more customization options to tailor Copilot's behavior and more control over the code content generated by Copilot.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • May combine two claims.

    Findingown data

    The most common causes of Copilot usage problems are Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue.

    “Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most common causes of these problems.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • May combine two claims.

    Findingown data

    The predominant solutions to Copilot usage problems are Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version.

    “Bug Fixed by Copilot, Modify Configuration/Setting and Use Suitable Version are the predominant solution.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • May combine two claims.

    Findingown data

    After manual screening, the study obtained 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts related to Copilot, yielding 1353 problems, 391 causes, and 497 solutions.

    “After manual screening, we obtained 473 GitHub issues, 706 GitHub discussions, and 142 SO posts related to Copilot and got a total of 1353 problems, 391 causes, and 497 solutions based on our data extraction criteria.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These facts have raised concern about copyright and intellectual property when Copilot is used, especially for in-house development.

    “These two facts have led people to pay more attention to copyright and intellectual property problems when using Copilot, especially in inhouse development.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Copilot is trained on a large corpus of open-source code.
    • Rests on an assumption: Copilot generates suggestions based on users' code context and collects that context during operation.

  • May combine two claims.

    Interpretationreasoning

    Compatibility Issue stems from the complex environments where users run Copilot and from Copilot's own compatibility robustness.

    “Compatibility Issue arises from the complex environments in which users utilize Copilot, as well as the compatibility robustness of Copilot itself.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Cohen's Kappa measured inter-rater reliability between the two authors at 0.824, 0.834, and 0.806, indicating reasonable agreement.

    “The inter-rater reliability between the two authors was measured by the Cohen's Kappa coefficient (Cohen, 1960), resulting in values of 0.824, 0.834, and 0.806, which indicate a reasonable level of agreement between the two authors.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cohen, 1960 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The second and third authors verified the initial analysis results, and conflicts were resolved using a negotiated agreement approach.

    “3) The initial version of the analysis results was further verified by the second and third authors, and the negotiated agreement approach (Campbell et al., 2013) was employed to address the conflicts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Campbell et al., 2013 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Any labelling disagreements were discussed with the second author until a consensus was reached.

    “For any discrepancies in the results, the two authors engaged in discussions with the second author to reach a consensus.”
    • Also stated in: § 2.3.2. Formal Data Labelling · plain
      “During the labelling process, any result on which the two authors disagreed was subject to discussion with the second author until an agreement was reached.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The absence of Copilot version information in the dataset prevented a thorough consideration of version effects.

    “However, due to the absence of Copilot version information in the dataset, we could not consider the version information thoroughly in this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Open Coding and Constant Comparison are widely used Grounded Theory techniques for qualitative data analysis.

    “we conducted data analysis by using the Open Coding and Constant Comparison methods, which are two widely employed techniques from Grounded Theory during qualitative data analysis (Stol et al., 2016).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Stol et al., 2016

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Copilot now supports HTTP proxy access, which resolves such problems.

    “However, Copilot now offers support for access through an HTTP proxy, thus addressing such problems (GitHub, 2024b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites GitHub, 2024b

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GitHub Copilot was one of the most popular AI-assisted coding tools, with 1.3 million paid users by February 2024.

    “GitHub Copilot is one of the most popular AI-assisted coding tools and has been widely used for software development with 1.3 million paid users till Feb of 2024 (Wilkinson, 2024).”
    • Also stated in: § 1. Introduction · plain
      “Since its release, Copilot has gained significant attention within the developer community, and it had a total of 1.3 million paid users till Feb of 2024 (Wilkinson, 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wilkinson, 2024

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A pilot data labelling was conducted by the first and third authors to reduce personal bias.

    “To minimize personal bias in the formal labelling process, the first and third authors conducted a pilot data labelling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    PLUG-IN COMPATIBILITY ISSUE is less predictable and often occurs when Copilot is used with other code completion tools.

    “The appearance of PLUG-IN COMPATIBILITY ISSUE (70) is less predictable, which often arises when using Copilot with other code completion tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The pilot extraction showed that the three data items were extractable from the dataset.

    “The results indicated that the three data items could be extracted from our dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GitHub Issues, GitHub Discussions, and Stack Overflow have been frequently used in empirical software engineering studies to collect developer-reported problems.

    “Data from these three sources has been frequently used in empirical studies to gather various developer-reported problems in software engineering (Bird et al., 2023) (de Dieu et al., 2023) (El aoun et al., 2021).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Bird et al., 2023 source

      Not checked in the free run.

    • cites de Dieu et al., 2023

      Not checked in the free run.

    • cites El aoun et al., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    A pilot data extraction was performed on 20 randomly selected GitHub issues, 20 discussions, and 20 Stack Overflow posts, with the second author resolving discrepancies.

    “The first and third author conducted a pilot data extraction on 20 randomly selected GitHub issues, 20 discussions, and 20 SO posts, and in case of any discrepancies, the second author was involved to reach a consensus.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    A better user experience can result from using IDEs or code editors that Copilot officially supports.

    “IDEs or code editors that are officially supported by Copilot can lead to a better user experience.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.
    • Rests on an assumption: The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study gathered data from three platforms-GitHub Issues, GitHub Discussions, and Stack Overflow-to make the dataset comprehensive.

    “Besides, we collected data from three popular software development platforms and forums, i.e., GitHub Issues, GitHub Discussions, and SO, to ensure the comprehensiveness of our dataset.”
    Quote in the paperNot a substantial claim
32 of 32

  • The paper relies on this without support.

    Empirical evidence about practitioners' actual experiences with Copilot is scarce.

    “Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot.”

  • The paper relies on this without support.

    No systematic categorization exists for the problems developers face when using Copilot in practice, nor for their causes and solutions.

    “However, there is currently a lack of systematic categorization of the problems arise during the practical use of Copilot from the perspective of developers, as well as the causes behind them and solutions for addressing them.”

  • The paper relies on this without support.

    Copilot is trained on a large corpus of open-source code.

    “Copilot is trained on a large corpus of open source code and generates code suggestions based on the users' code context.”

  • The paper relies on this without support.

    Copilot generates suggestions based on users' code context and collects that context during operation.

    “Copilot is trained on multi-language open-source code and also needs to collect users' code context during its operation to generate suggestions.”

  • The paper relies on this without support.

    The outputs of large language models are unpredictable.

    “Since the outputs of LLMs are unpredictable”

  • The paper relies on this without support.

    The code snippets used to train Copilot potentially contain quality issues.

    “the code snippets used to train Copilot potentially contain quality issues”

  • The paper relies on this without support.

    The problems, causes, and solutions extracted from GitHub Issues, GitHub Discussions, and Stack Overflow are representative of the real-world problems Copilot users encounter.

    “Our research is grounded in the perspective of software developers, focusing on the real-world problems they encounter when using Copilot”
    “we collected data from three popular software development platforms and forums, i.e., GitHub Issues, GitHub Discussions, and SO, to ensure the comprehensiveness of our dataset.”

  • The paper relies on this without support.

    The manual screening and data extraction criteria reliably identify problems, causes, and solutions.

    “After manual screening, we obtained 473 GitHub issues, 706 GitHub discussions, and 142 SO posts related to Copilot and got a total of 1353 problems, 391 causes, and 497 solutions based on our data extraction criteria.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The existing studies do not consider individual activities within the software development life-cycle and their specificities.

    “However, the existing studies do not consider individual activities within the software development life-cycle and their specificities.”

    From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [12]

  • Addressing AI-generated output being inaccurate and Lack of compliance with non-functional requirements is crucial, with 17.7% and 3.2% of all responses mentioning these issues, respectively.

    “Addressing AI-generated output being inaccurate and the Lack of compliance with non-functional requirements is crucial, with 17.7% and 3.2% of all responses mentioning these issues, respectively, which is in line with the study of Liang et al. [12].”

    From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [12]

  • The results of the study support the existing notion that the current state of affairs of using AI assistants in software development is favorable and rapidly developing.

    “The results of our study support the existing notion about the current state of affairs of using AI assistants in software development as favorable and rapidly developing [12,15].”

    From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure and text disagree on the number of labelled issues

    Minor

    The research-process figure lists 474 issues for data labelling, whereas the formal labelling results list 473 GitHub issues.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M.
Affiliations
  • Wuhan University (CN)
  • Central China Normal University (CN)
  • Lancaster University (GB)
  • RMIT University (AU)
  • University of Jyväskylä (FI)
Corresponding author
Liang, Peng
ORCID
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2025-01
Volume
219
Pages
112204
ISSN
0164-1212 (ISSN-L)
DOI
10.1016/j.jss.2024.112204
OpenAlex
W4402129912
Cited by
30
References
46
Citations by year
2026: 15 · 2025: 14 · 2024: 1
FWCI
11.28
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Systems and Software (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Lancaster EPrints (Lancaster University) (submittedVersion)
  • Jyväskylä University Digital Archive (University of Jyväskylä) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Funders
National Natural Science Foundation of China
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
GitHub Copilot, large language models, AI-assisted code generation, developer experience, compatibility issues, bug fixes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review