CoChat
Sign Up
Cited by 128Software Engineering ResearchWeb Application Security VulnerabilitiesSoftware Testing and Debugging Techniques

Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?

Open access
Asare, Owura, Nagappan, Meiyappan, Asokan, N.·2023·Empirical Software Engineering
DOI 10.1007/s10664-023-10380-1
Open this paper in CoChat — free accountOpen PDF

AI snapshot

Written from the paper’s full text.

TL;DR

This paper compares Copilot-generated code to human-written vulnerable code using C/C++ vulnerability scenarios from Big-Vul. It concludes that Copilot is not as bad as human developers at introducing vulnerabilities, though its behavior varies by vulnerability type and age.

Key findings

  • After recategorization, Copilot reproduced the original vulnerable code in 51/153 cases (33.3%) and reproduced the human fix in 39/153 cases (25.5%); 63/153 outputs (41.2%) remained Category C.
  • Preliminary exact-match categorization found 35/153 (22.9%) Category A, 31/153 (20.3%) Category B, and 87/153 (56.8%) Category C; 24 of the 87 Category C samples were recategorized, with 16 into A and 8 into B.
  • Copilot was more likely to generate the fix for more recent vulnerabilities: Category B rose from 15.7% in 2017 to 25.7% in 2018 and 40.6% in 2019, while Category C fell from 52.9% to 40.0% to 25.0%.
  • Vulnerability type mattered: CWE-20 (improper input validation) yielded Category A 100% of the time, while CWE-119, CWE-190, and CWE-476 were more likely to yield Category B; CWE-20 had infinite Category A affinity, CWE-666 1.4, CWE-119 0.88, and CWE-190/CWE-476 0.67.

Methods

  • Used the Big-Vul dataset of 3,754 C/C++ vulnerabilities across 348 projects from 2002–2019, containing 4,432 fix commits; filtered to 2,226 single-file, single-contiguous-location samples, then selected 153 most recent samples for evaluation.
  • Re-created pre-vulnerability prompts from buggy/fixed files and collected Copilot’s top suggestion in VS Code, using Atom for file editing to prevent Copilot from peeking; samples where Copilot only suggested comments were excluded.
  • Categorized outputs in two stages: exact text matches to vulnerable code (A) or fixed code (B), then three independent C/C++-experienced coders recategorized Category C outputs when at least two agreed the code was compilable and belonged to A or B.

Limitations

  • The paper states that manual inspection may miss other vulnerabilities, that about 42% of samples remained Category C with unknown vulnerability level, and that it cannot determine whether other vulnerability types exist in Category A/B samples.
  • The paper states that prompt recreation cannot account for external files or all context available to human developers, that Copilot may have replicated training data, and that lack of training-data access limits explanation of memorization.
  • The paper states that CWE counts were uneven, findings are limited to C/C++ and Copilot, and Copilot’s non-deterministic/autopilot use may not generalize to assistant-style use or other code generation tools.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10664-023-10380-1

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § 1 Introduction · p. 3

    Copilot is more likely to generate the fix for a vulnerability when the vulnerability is more easily avoidable.

    “and when the vulnerability is more easily avoidable (Section 5.3).”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 6.1 Construct Validity · p. 19

    The analysis resulted in a substantial number of category C samples (approximately 42%).

    “Our analysis resulted in a substantial number of category C samples ( ≈ 42%).”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 5.4 Implications for Automated Vulnerability Fixing · p. 17

    Category A was larger than Category B in the experiments.

    “since Category A was larger than Category B in our experiments.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 5 Results and Discussion · p. 13

    Copilot generated the same vulnerability as human developers in 51 of 153 cases (33.3%).

    “it generated the same vulnerability as humans only in 51/153 cases (33.3%)”
    • Also stated in: § Abstract · p. 1 · plain
      “We find that Copilot replicates the original vulnerable code about 33% of the time”
    Quote in the paperSubstantial claim
  • Findingown data
    § 5.1 Results Overview · p. 13

    In preliminary exact-match categorization, Copilot reproduced the corresponding fix in 31 of 153 cases (20.3%).

    “In 31 cases (20.3%), Copilot reproduced the corresponding fix for the original vulnerability (Category B).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 23
  • The quote only partly states this claim. 1
  • May combine two claims. 18

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data
    § 4.3 Scenario Re-creation · p. 9

    The study evaluated 153 samples.

    “we capped our sample size at 153.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 5.1 Results Overview · p. 14

    The study covered 28 of the 78 CWEs in the original Big-Vul dataset.

    “Overall, we covered 28 of the 78 CWEs in the original Big-Vul dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Results Overview · p. 13

    Recategorization moved 24 of the 87 Category C samples (approximately 28%) into Category A or B, with 16 into A and 8 into B.

    “The recategorization of category C outputs as described in section 4.7 resulted in 24 of the 87 (approximately 28%) category C samples being recategorized, with 16 going into category A and 8 going into category B.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 5.1 Results Overview · p. 14

    After recategorization, the final effective totals were 51 samples in Category A, 39 in Category B, and 63 in Category C.

    “Taking the recategorization into account, the effective total of samples in each category are 51, 39, and 63 for A, B, and C respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Code Replication · p. 15

    Over the years, as the proportion of category C samples decreased, there was a corresponding increase in the proportion of category B outputs while the proportion of category A outputs remained relatively constant.

    “We saw that over the years, as the proportion of category C samples decreased, there was a corresponding increase in the proportion of category B outputs while the proportion of category A outputs remained relatively constant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Code Replication · p. 15

    The study found no definitive evidence of memorization or strong preference for category A or B suggestions.

    “Overall, while we found no definitive evidence of memorization or strong preference for category A or B suggestions”
    • Also stated in: § 6.2 Internal Validity · p. 20 · plain
      “Copilot's lack of preference for either the vulnerable or vulnerability-fixing code”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2 Dataset Preprocessing · p. 8

    The filtration of Big-Vul yielded a subset of 2,226 samples from the original 4,432 samples.

    “The filtration yielded a subset with 2,226 samples from an original set of 4,432 samples.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.7 Recategorization of Category C Outputs · p. 11

    Outputs were recategorized only when at least two of three coders agreed that the code was compilable and could belong to one of the other two categories.

    “Outputs were recategorized only when at least two of the three coders agreed that (1) the code was compilable and (2) the code could belong to one of the other two categories.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 4.4 Preventing Copilot Peeking · p. 10

    To prevent Copilot from accessing code before output generation, initial file creation and editing during scenario re-creation was performed in Atom, a different text editor from VS Code.

    “To prevent such undesired Copilot access to code before the output generation step, the initial creation and editing of files during the scenario re-creation stage was performed in a different text editor (Atom).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 4.6 Preliminary Categorization of Outputs · p. 11

    The study limited categorization to whether Copilot-generated code consisted of the original human-induced vulnerability or the subsequent human-developed fix, and did not attempt to identify whether other vulnerabilities were introduced by Copilot.

    “We limited our categorization to determining whether the Copilot-generated code consisted of the original human-induced vulnerability or the subsequent human-developed fix. We did not attempt to identify whether other vulnerabilities were introduced by Copilot.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.2 Code Replication · p. 14

    Deep-learning based code generation tools have a tendency to clone training data.

    “From previous studies (Ciniselli et al., 2022), we know that deep-learning based CGTs have a tendency to clone training data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ciniselli et al., 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § 4.3 Scenario Re-creation · p. 9

    The study was registered as a registered report with a goal to evaluate at least 100 samples.

    “In our published registered report (Asare et al., 2022), we indicated that our goal was to evaluate at least 100 samples.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Asare et al., 2022

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning
    § 1 Introduction · p. 2

    Code used to train CGTs is obtained from a variety of sources and usually has no guarantee of being secure.

    “Code used to train such models is obtained from a variety of sources and usually has no guarantee of being secure.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 4.6 Preliminary Categorization of Outputs · p. 11

    Three independent coders manually examined each Category C sample to recategorize it.

    “We then asked three independent coders to manually examine each sample in Category C to see if they could be recategorized into one of the other two categories based on the coder's expertise.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning
    § 1 Introduction · p. 2

    The primary objective of a CGT is to generate the most likely continuation for a given prompt; other concerns such as security may not yet be a priority.

    “This is because the primary objective of a CGT is to generate the most likely continuation for a given prompt; other concerns such as the level of security of the output may not yet be a priority.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § 1 Introduction · p. 2

    Even when a CGT is trained on code considered secure today, vulnerabilities may be discovered in that code in the future.

    “Even when a CGT is trained on code considered secure today, vulnerabilities may be discovered in that code in the future.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    81% of 2,049 human-developer codebases contain at least one vulnerability, and 49% contain at least one high-risk vulnerability.

    “according to the 2022 Open Source Security and Risk Analysis (OSSRA) report by Synopsys, 81% of (2,049) codebases (of human developers) contain at least one vulnerability while 49% contain at least one high-risk vulnerability (Synopsys, 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Synopsys, 2022

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.7 Recategorization of Category C Outputs · p. 12

    The coders were not informed whether an image contained buggy or fixed code.

    “The coders were not informed whether an image contained buggy or fixed code.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 4.1 Dataset · p. 6

    The Big-Vul dataset consists of 3,754 C/C++ vulnerabilities across 348 projects from 2002 to 2019, with 4,432 samples representing commits that fix vulnerabilities.

    “The dataset consists of a total of 3,754 C and C++ vulnerabilities across 348 projects from 2002 and 2019. There are 4,432 samples in the dataset that represent commits that fix vulnerabilities in a project.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fan et al., 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation
    § 5.3 Vulnerability Analysis · p. 16

    The study's findings are in line with Pearce et al. (2022), who also show that Copilot has varied security performance depending on the kinds of vulnerabilities it is presented with.

    “Broadly speaking, our findings here are in line with the findings by Pearce et al. (Pearce et al., 2022) who also show that Copilot has varied performance, security-wise, depending on the kinds of vulnerabilities it is presented with.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Pearce et al., 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.4 Implications for Automated Vulnerability Fixing · p. 17

    Recent research has shown that CGTs based on language models possess some ability for zero-shot vulnerability fixing.

    “Recent research has shown that CGTs based on language models possess some ability for zero-shot vulnerability fixing (Pearce et al., 2023; Prenner et al., 2022; Zhang et al., 2022; Jiang et al., 2021).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Pearce et al., 2023

      Not checked in the free run.

    • cites Prenner et al., 2022

      Not checked in the free run.

    • cites Zhang et al., 2022

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given
    § 6.1 Construct Validity · p. 19

    The re-creation process attempts to mimic the sequential order in which developers write code within a file, but it is unable to take into account other external files that the developer might have known about.

    “While our re-creation process attempts to mimic the sequential order in which developers write code within a file, we are unable to take into account other external files that the developer might have known about.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5.2 Code Replication · p. 14

    According to GitHub, Copilot copies code snippets longer than 150 characters from its training data only 1% of the time.

    “their internal research found that Copilot copies code snippets (from its training data) with longer than 150 characters only 1% of the time (GitHub Inc., 2021).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites GitHub Inc., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 4.3 Scenario Re-creation · p. 9

    The sample size of 153 gives a 90% confidence level with a 7% margin of error.

    “The sample size of 153 gives us a 90% confidence level with a 7% margin of error.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § 4.7 Recategorization of Category C Outputs · p. 12

    The coders worked independently and their final responses were aggregated by the authors.

    “They worked independently and the final responses were aggregated by the authors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 4.6 Preliminary Categorization of Outputs · p. 11

    The task of identifying vulnerabilities in source code is an open research problem, and automated vulnerability detection has high false positive and negative rates.

    “This was because the task of identifying vulnerabilities in source code was and still is an open research problem. While there have been some attempts to address automated vulnerability detection (GitHub Inc., 2019; ? ), false positive and negative rates remain high (Chakraborty et al., 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Chakraborty et al., 2022

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data
    § 5.4 Implications for Automated Vulnerability Fixing · p. 17

    Copilot (in its current state) is less likely to generate a fix and more likely to reintroduce a vulnerability.

    “Our findings suggest that Copilot (in its current state) is less likely to generate a fix and more likely to reintroduce a vulnerability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given
    § 1 Introduction · p. 2

    Human developers are not always able to write secure code.

    “This is because the code was likely written by humans who are themselves not always able to write secure code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § 4.7 Recategorization of Category C Outputs · p. 12

    The coders were graduate students in computer science with at least 4 years of C/C++ development experience.

    “The coders were graduate students, from the CS department at the University of Waterloo, with at least 4 years of C/C++ development experience.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 6.3 External Validity · p. 20

    The findings are still valid for C/C++, which are still widely used.

    “The findings are still valid for C/C++ which are still widely used.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 6.3 External Validity · p. 20

    The dataset used for this evaluation only contained C and C++ code, meaning Copilot's performance in this study may not generalize to significantly different programming languages.

    “The dataset used for this evaluation only contained C and C++ code meaning Copilot's performance in this study may not generalize to significantly different programming languages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.3 External Validity · p. 20

    Due to the diverse nature in which CGTs like Copilot can be prompted, combined with their non-deterministic nature, it is difficult to assume that Copilot's performance with respect to the different vulnerabilities will always be the same.

    “Due to the diverse nature in which CGTs like Copilot can be prompted, combined with their non-deterministic nature, it is difficult to assume that Copilot's performance with respect to the different vulnerabilities will always be the same.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § 6.3 External Validity · p. 20

    Critiques of and improvements to Copilot will probably apply to other CGTs as well, considering that Copilot is a fairly advanced and widely popular tool.

    “However, considering that Copilot is a fairly advanced and widely popular tool, we believe that critiques of and improvements to Copilot will probably apply to other CGTs as well.”
    Quote in the paperNot a substantial claim
33 of 33

  • The paper relies on this without support.

    The Big-Vul dataset's vulnerability scenarios are representative of human developers' vulnerability introduction behavior in general.

    “We present Copilot with the same cases and analyze its performance by inspecting and categorizing its output based on whether it introduces the same (or similar) vulnerability or not.”

  • The paper relies on this without support.

    The CVE publish date is a valid proxy for the age of the vulnerability.

    “The 2,226 samples were sorted by publish date (most recently published vulnerabilities first) and used in the scenario re-creation stage.”
    “we observe a trend of Copilot being more likely to generate the fix for a vulnerability when the sample has a more recent publish date (Section 5.2)”

  • The paper relies on this without support.

    The original vulnerable code in the Big-Vul dataset was introduced by human developers.

    “Our evaluation uses a dataset of vulnerabilities to investigate whether Copilot will always perform as badly as human developers in cases where the latter have previously introduced vulnerable code.”

  • The paper relies on this without support.

    The Big-Vul dataset's vulnerabilities are representative of real-world vulnerabilities.

    “This study focuses on Copilot's security performance on a dataset of real world vulnerabilities.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Security vulnerabilities and possible overreliance issues are important areas of research that we do not address in this paper.

    “To this end, security vulnerabilities and possible overreliance issues [2,25,26], are important areas of research that we do not address in this paper.”

    From Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming (2024) · cites this paper as [2]

    Section 3.1 (p. 6) calls the security of code generated by assistants an issue requiring closer attention, and the Results (p. 13) examine Copilot-generated vulnerabilities. The cited paper does not establish possible overreliance issues; the statement that the citing authors do not address these topics is their own description of their paper. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLM-based coding assistants can generate code containing common weaknesses such as SQL injection, path traversal, command injection, insecure cryptographic usage, and unsafe resource handling.

    “Prior studies show that LLM-based coding assistants can generate code containing common weaknesses such as SQL injection, path traversal, command injection, insecure cryptographic usage, and unsafe resource handling [2,3,4].”

    From SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software (2026) · cites this paper as [3]

    The Abstract (p. 1) reports that Copilot reproduced vulnerable code, and Table 8 (p. 18) includes resource-management errors among the weaknesses examined. The cited paper does not establish the claim’s other named examples—SQL injection, path traversal, command injection, or insecure cryptographic usage—as weaknesses generated by Copilot. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Recent research highlights security vulnerabilities associated with code generated by LLMs.

    “Recent research highlights the security vulnerabilities associated with code generated by LLMs [2, 6, 15, 17, 20, 31, 32, 36, 38, 41, 49].”

    From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [2]

  • Asare et al. found that while the GenAI tool introduced security vulnerabilities, the rates were not higher than those introduced by human developers.

    “Asare et al. [43] compared vulnerability rates between human-written and Copilot-generated code and found that, while the GenAI tool introduced security vulnerabilities, the rates were not higher than those introduced by human developers.”

    From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [43]

  • In line with previous studies, people are cautious regarding AI-generated code being insecure.

    “However, in line with previous studies [30,31,32], we see that people are cautious regarding AI-generated code being insecure.”

    From Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward (2025) · cites this paper as [31]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • CWE-476 caption cites the CWE-20 buggy response

    Minor

    The CWE-476 figure caption identifies listing 8 as its Copilot output, but listing 8 is identified elsewhere as the buggy Category A response for CWE-20. The CWE-476 response is labeled listing 12.

  • Sample selection is described as both random and date-ordered

    Minor

    The methodology says samples were selected by repeatedly taking the most recent one, whereas the results section says they were selected randomly.

2 of 2

The citation lists haven’t been fetched yet

References 31 · Cited by 128

Research works printed but not linked: 5

  1. Bengio, Y., R. Ducharme, and P. Vincent 2000.A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems,Volume 13. MIT Press
  2. Hardmeier, C. 2016, December.A Neural Model for Part-of-Speech Tagging in Historical Texts.In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, pp.922–931. The COLING 2016 Organizing Committee
  3. Le, T.H.M., H. Chen, and M.A. Babar. 2020, June.Deep Learning for Source Code Modeling and Generation:Models, Applications, and Challenges. ACM Comput. Surv. 53(3)https://doi.org/10.1162/neco.10.1145/3383458 · DOI
  4. Nijkamp, E., B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2022.CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.arXiv preprint
  5. Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł Kaiser, and I. Polosukhin 2017.Attention is All You Need.In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA,pp.6000–6010. Curran Associates Inc.event-place: Long Beach, California, USA

Authors
Asare, O., Nagappan, M., & Asokan, N.
Affiliations
University of Waterloo (CA)
Corresponding author
Asare, Owura
ORCID
Venue
Empirical Software Engineering
Publisher
Springer Science and Business Media LLC
Published
2023-09-23
Volume
28
Issue
6
ISSN
  • 1382-3256 (ISSN-L)
  • 1573-7616
DOI
10.1007/s10664-023-10380-1
OpenAlex
W4386982649
Cited by
128
References
31
Citations by year
2026: 32 · 2025: 42 · 2024: 37 · 2023: 16 · 2022: 1
FWCI
29.80
Type
journal article
Language
en
Open access
Licence
Locations
  • Empirical Software Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
WHJIL
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
GitHub Copilot, AI-assisted code generation, software vulnerabilities, C/C++ vulnerabilities, neural language models, developer productivity, vulnerability types

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review