Asleep at the keyboard? Assessing the security of GitHub copilot’s code contributions

May combine two claims.

The scenarios with the highest mean vulnerable scores were 22-0 (0.83), 125-1 (0.76), and 787-0 (0.74), with no overlap with the top three non-vulnerable scenarios.

“If we instead consider the scenarios with the highest mean vulnerable scores, the scenarios are 22-0 (0.83), 125-1 (0.76), and 787-0 (0.74)-with no crossover in the top 3.”

May combine two claims.

Most scenarios had similar top-answer scores, but scenarios 476-1 (0.96), 200-0 (0.93), and 416-2 (0.92) had very high-confidence yet vulnerable top-scoring options.

“Most scenarios featured similar scoring top answers, although a few stand out: 476-1 (at 0.96), 200-0 (0.93), and 416-2 (0.92) all had an extremely high-confidence but vulnerable top-scoring option.”

May combine two claims.

Although both CWE-79 and CWE-22 scenarios were evaluated for C and Python, CWE-79 had no vulnerable top-scoring programs and only 19% vulnerable options overall, whereas CWE-22 had only vulnerable top-scoring programs and 60% vulnerable options overall.

“Both scenarios evaluated programs for both C and Python, yet CWE-79 had zero vulnerable top-scoring programs, and only 19 % vulnerable options overall, while CWE-22 had only vulnerable top-scoring programs, with 60 % vulnerable options overall.”

May combine two claims.

Scenario C-3 made that added function vulnerable; confidence rose markedly, but the answers skewed toward vulnerable, with only one non-vulnerable answer, and the top-scoring option was vulnerable.

“C-3, meanwhile, made that new function vulnerable.This also increased the confidence markedly, but the answers are now skewed towards vulnerable-only one non-vulnerable answer was generated.The top-scoring option is vulnerable.”

May combine two claims.

In scenario 787-0, sprintf can produce strings up to 317 characters from %f, requiring buffers of at least 48 characters, but the buffers are only 32 characters, so sprintf may write beyond the buffer.

“This is because sprintf may generate strings up to 317 characters from %f (317 for doubles, 47 for floats).This means that these character buffers must be at least 48 characters (these are floats, and the code also needs to include space for the null termination character).Yet, each buffer is only 32 characters long, meaning that sprintf may write past end of the buffer.”

Classified as probably not a substantial claim.

The study designed 54 scenarios covering 18 different CWEs.

“In total, we designed 54 scenarios across 18 different CWEs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There were 25 C scenarios, which produced 513 programs.

“Breaking down by language, 25 scenarios were in C, generating 513 programs.”

May combine two claims.

Copilot's confidence and performance stayed close to CON overall, except for notable differences in scenarios C-2 and C-3.

“Copilot did not diverge far from the overall confidences and performance of CON, with two notable exceptions in C-2 and C-3.”

May combine two claims.

Compared with Python and C, Copilot struggled to generate meaningful Verilog; scenario 1254-2 produced no syntactically compliant results.

“Compared with Python and C, Copilot struggled to generate meaningful Verilog (for example, 1254-2 had no syntactically compliant results).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In scenario 79-0, Copilot's top answer was marked non-vulnerable because the username might be escaped inside render_template().

“As this code is not definitively vulnerable (the username may be escaped inside render_template(), it is marked as non-vulnerable.”

May combine two claims.

For scenario 787-0, Copilot's highest-scoring suggestion had a rounded confidence score of 0.841 and was classified as vulnerable by CodeQL.

“The top option for this from Copilot, with a rounded score of 0.841, is presented in Figure 4(b).Here, Copilot's generated code is marked as vulnerable by CodeQL.”

May combine two claims.

The CON control prompt produced 6 vulnerable suggestions and 19 non-vulnerable suggestions, with a non-vulnerable top suggestion.

“It had 6 vulnerable suggestions, 19 non-vulnerable, and the top suggested option non-vulnerable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There were 29 Python scenarios, producing 571 programs in total.

“29 scenarios were in Python, generating 571 programs total.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In scenario 79-0, the third-scoring option was definitively vulnerable, while the fourth-highest option was not vulnerable.

“The third-scoring option, presented in Figure 5(c), is definitively vulnerable, and the fourth-highest, Figure 5(d), is not vulnerable.”

Classified as probably not a substantial claim. May combine two claims.

Verilog syntax resembles other C-type languages, and many non-compiling options used keywords and syntax from those languages, especially SystemVerilog.

“Verilog has syntax similar to other C-type languages.Many of the non-compiling options used keywords and syntax from these other languages, particularly SystemVerilog.”

May combine two claims.

The top-scoring non-vulnerable options were for 732-2 (0.91), 306-2 (0.91), and 125-1 (0.90); the highest mean non-vulnerable scores were for 306-2 (0.82), 416-2 (0.78), and 79-1 (0.76).

“Of the non-vulnerable options, the top-scoring were for 732-2 (0.91), 306-2 (0.91), and 125-1 (0.90), and the scenarios with the highest mean non-vulnerable scores were 306-2 (0.82), 416-2 (0.78), and 79-1 (0.76).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CWE-200 is a higher-level, context-required weakness about information leakage and showed a wide range of confidence scores.

“Meanwhile, CWE-200, which is a higher-level context-required CWE concerning information leakage, had a wide Figure 5. Scenario 79-0 range of confidences.”

May combine two claims.

For hardware CWEs, Copilot generated 198 programs; 56 (28.28%) were vulnerable, and 7 of 18 scenarios (38.89%) had vulnerable top-scoring options.

“Copilot was able to generate options to make 198 programs.Of these, 56 (28.28 %) were vulnerable.Of the 18 scenarios, 7 (38.89%) had vulnerable top-scoring options.”

May combine two claims.

Scenario C-2 added a separate non-vulnerable database function; this greatly improved Copilot's output, raised the confidence score, and appeared to eliminate other vulnerable suggestions.

“Here, C-2 adds a separate non-vulnerable database function to the program.This function is non-vulnerable.This significantly improved the output of Copilot, with an increase in the confidence score, and seemed to completely prevent any other vulnerable suggestions.”

May combine two claims.

The study generated 89 scenarios for Copilot, which produced 1,689 programs.

“In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs.”

The quote only partly states this claim.

13 of the 25 C scenarios (52.00%) had a top-scoring program that was vulnerable.

“Of the scenarios, 13 (52.00%) had a top-scoring program vulnerable.”

May combine two claims.

For the six hardware CWEs, the authors sought insecure rather than correct code, and Copilot performed relatively well in that respect.

“For the six CWEs we were not looking for correct code, but rather for insecure code.In this regard, Copilot performed relatively well.”

May combine two claims.

Across all axes and languages, 39.33% of top options and 40.73% of total options were vulnerable.

“across all axes and languages, 39.33 % of the top and 40.73 % of the total options were vulnerable”

May combine two claims.

The prompt diversity experiment covered 17 scenarios and 407 valid Copilot-generated programs; 152 (37.35%) were vulnerable, and 4 scenarios (25.53%) had vulnerable top-scoring programs.

“Overall, we collected results for 17 different scenarios, with Copilot options generating 407 valid programs.Of these, 152 (37.35 %) were vulnerable.Across the 17 scenarios, 4 (25.53 %) had top-scoring vulnerable programs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Subtle prompt variations affect Copilot's code generation, and wider contexts with better-quality code can yield more secure suggestions.

“Subtle variations in the prompts (Section 5.3) affect Copilot's code generation; wider contexts with better quality code can yield more secure code suggestions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

CWE-79 is MITRE's second-highest ranked weakness and concerns web applications failing to neutralize user-provided values before including them in outputs.

“At MITRE's second-highest rank, CWE-79 is a weakness that covers those scenarios where web applications do not neutralize user-provided values before including them in their web application outputs.”

Classified as probably not a substantial claim.

The evaluation assessed only vulnerabilities, not functional correctness.

“In addition, we do not evaluate for correctness, only for vulnerabilities.”

• Also stated in: § 5.1.1 Vulnerability Classification · plain

  “Finally, we did not evaluate for functionally correct code generation, only vulnerable outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hardware CWEs differ from software CWEs mainly in requiring external context (assets) beyond the hardware definition, including implementation and timing details.

“Hardware CWEs have some key differences from software CWEs, primarily surrounding external context (assets) beyond what is provided with the hardware definition directly, 6 including implementation and timing details.”

• cites [6] 6. Chen, M. et al. Evaluating large language models trained on code. arXiv:2107.03374 [cs], July 2021, arXiv: 5. 2107.03374.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The security of top options matters especially because novice users may be more likely to accept the best suggestion.

“The security of the top options are particularly important-novice users may be more likely to accept the 'best' suggestion.”

• AssumptionRests on an assumption: Novice users are more likely to accept Copilot's top suggestion.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hardware CWEs were added to MITRE's taxonomy in 2020.

“Hardware CWEs were added to MITRE's taxonomy in 2020.23”

• cites [23] 23. Reddy, S., Chen, D., and Manning, C.D. CoQA: A conversational question answering challenge. Transactions of 19. the Association for Computational Linguistics 7, (2019), 249-266.Not checked yet

  Not checked in the free run.

May combine two claims.

CWE-787 was the most severe weakness in the corpus, ranked first because memory errors are common in low-level code and exploitable writable buffer errors can cause system compromise and arbitrary code execution.

“The most severe vulnerability in the corpus was CWE-787, or 'Out-of-bounds Write', which concerns mismanagement of low-level memory (specifically, when software may write data past the end or before the beginning of a buffer).This is ranked as #1 for two reasons: firstly, memory errors are frequently found in low-level code, and secondly, when exploitable, writable memory buffer errors can lead to”

Probably not a substantial claim: may not pass S3 (load-bearing).

CWE-476 and CWE-416 both involve low-level pointer mismanagement errors.

“These have some similarities between them, with CWE-476 and CWE-416 both deal with low-level pointer mismanagement errors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CWE-732 and CWE-306 are more qualitative and concern permissions and authorization, while CWE-125 concerns buffer over- and under-reads.

“Here, CWE-732 and CWE-306 are more qualitative and are concerned with permissions and authorization.Meanwhile, CWE-125 is for buffer over-and under-reads.”

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

The presence of either vulnerable or non-vulnerable SQL in a codebase is the strongest predictor of whether other vulnerable SQL exists there, and thus has the strongest influence on whether Copilot generates SQL vulnerable to injection.

“We hypothesize that the presence of either vulnerable or non-vulnerable SQL in a codebase is, therefore, the strongest predictor of whether or not there would be other vulnerable SQL in the codebase, and therefore, has the strongest impact upon whether or not Copilot will itself generate SQL code vulnerable to injection.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Copilot's variable security quality likely comes from the nature of community-provided open-source code: bugs that are more visible in open-source repositories are more often reproduced by Copilot.

“As Copilot is trained over open-source code available on GitHub, we theorize that the variable security quality stems from the nature of the community-provided code.That is, where bugs are more visible in open-source repositories, those bugs will be more often reproduced by Copilot.”

• AssumptionRests on an assumption: Copilot is trained on open-source code available on GitHub.
• AssumptionRests on an assumption: Bugs are more visible in open-source repositories.

Probably not a substantial claim: may not pass S3 (load-bearing).

Earlier research applied CodeQL to find vulnerable commits in a JavaScript project.

“Prior work used CodeQL to identify vulnerable code commits in the life of a JavaScript project.3”

• cites [3] 3. Bandara, V. et al. Fix that fix commit: A real-world remediation analysis of javascript projects. In 2020 IEEE sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Copilot's security performance across the scenarios is mixed, given the large number of generated vulnerabilities.

“Copilot's response to our scenarios is mixed from a security standpoint, given the large number of generated vulnerabilities”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors believe Copilot's difficulty with Verilog is mostly due to the smaller amount of available training data.

“We think this is due mostly to the smaller amount of training data available.”

Classified as probably not a substantial claim. May combine two claims.

Each scenario started without the weakness; whether the final program was vulnerable depended solely on Copilot's completion.

“Note that each scenario does not contain the weakness from the outset; it is Copilot's completion that determines if the final program is vulnerable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Copilot is based on OpenAI Codex models, which are GPT-3 models adapted to code from GitHub.

“Copilot builds on the OpenAI Codex family of models 5 that are GPT-3 models 4 fine-tuned on code from GitHub.”

• cites [5] 5. Brown, T.B. et al. Language models are few-shot learners. arXiv:2005.14165 [cs], July 2020, arXiv: 2005.14165.Not checked yet

  Not checked in the free run.

• cites [4] 4. 20th Intern. Working Conf. on Source Code Analysis and Manipulation (SCAM), Sept. 2020, 198-202.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

CodeQL was set to check only for the CWE that each scenario targeted.

“Importantly, CodeQL is configured in this step to only examine for the specific CWE this scenario is designed for.”

May combine two claims.

Copilot is non-deterministic: for the same prompt it can produce different answers at different times, so its outputs are not directly reproducible.

“However, Copilot is non-deterministic, and other outputs are possible.”

• Also stated in: § 6.1.3 Reproducible Code Generation · plain

  “As a generative model, Copilot outputs are not directly reproducible.For the same given prompt, Copilot can generate different answers at different times.”

Classified as probably not a substantial claim.

Traditional RTL security verification combines formal verification and manual evaluation by security experts.

“Traditional security verification for RTL is a mix of formal verification and manual evaluation by security experts.9”

• cites [9] 9. Drechsler, R., Harris, I.G., and Wille, R. Generating formal system models from natural language descriptions.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Even with the complete codebase, CodeQL sometimes cannot parse important information; the authors found this for memory buffer sizes because CodeQL's ability to derive memory boundaries is limited.

“However, even with the complete codebase, CodeQL sometimes cannot parse important information.The authors found this to be the case when considering memory buffer sizes, as CodeQL's ability to derive memory boundaries (for example, array lengths) is limited in functionality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To make the research reproducible, the authors archived all options for every prompt.

“As such, to reproduce this research, we archived all options for every provided prompt.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is possible that across the full corpus CodeQL had edge cases where it failed safe by marking something vulnerable that was not.

“it is possible that across the full corpus of generated programs there may have been edge cases where CodeQL 'failed-safe', that is, marked something as vulnerable that was not.”

Classified as probably not a substantial claim.

The lack of a definition for Copilot's 'mean prob' score further hampers statistical claims.

“We are also further hampered in this by the lack of a definition for the 'mean prob' score that is returned by Copilot with each result.”

Classified as probably not a substantial claim.

Because Copilot is a black-box, closed-source service on a remote server, general users cannot directly examine the model that generates outputs.

“As Copilot is both a black-box and closed-source, residing on a remote server, general users (such as the authors of this paper) cannot directly examine the model used for generating outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For languages and scenarios CodeQL did not support, such as Verilog, CWEs were marked manually; the authors aimed for objective outputs using CWE definitions only, but human judgment may make individual results debatable.

“For the languages and scenarios that CodeQL did not support (for example, Verilog), the CWEs had to be marked manually.When marking manually, we strove for objective outputs, by considering the definitions of the relevant CWEs and nothing else.However, by introducing the human element, it is possible that individual results may be debatable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The scenarios are artificial because they target specific potential weaknesses; real-world code is messier and has more context, so the setup does not fully reflect real-world software.

“While scenarios provide insights into Copilot, the scenarios are artificial in that they try to target specific potential weaknesses.Real-world code is considerably messier and contains larger amounts of context (for example, other functions, comments, etc.), so our setup does not fully reflect the spectrum of real-world software.”

• AssumptionRests on an assumption: Real-world code is messier and contains more context than the scenarios.

Classified as probably not a substantial claim. May combine two claims.

The manual effort and rate-limiting of Copilot queries prevent efficient collection of large datasets, which shaped the methods used.

“The manual effort needed to query Copilot plus rate-limiting of queries, prohibits efficient collection of large datasets.This impacted and informed the methods we use.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors hoped the requested 25 options would include the corpus of possible answers, but this is not guaranteed because Copilot may be retrained on new repositories, and probing a proprietary black box risks future changes.

“Since we ask Copilot to generate a few lines of code, our hope was that the corpus of possible answers is included in the requested 25 options.However, this is not guaranteed, considering that Copilot may be re-trained over new code repositories at a later date-probing black-box proprietary systems has the risk that updates may render them different in future.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is difficult to assess statistical significance of all results, but the authors consider the empirical findings noteworthy.

“It is difficult to make claims on the statistical significance of all our results, but we believe that the empirical findings are nevertheless noteworthy.”