Do Users Write More Insecure Code with AI Assistants?

[21] is the only prior work conducting a controlled user study on security vulnerabilities in code written with AI assistance.

“To the best of our knowledge, [21] is the only work that conducts a controlled user study examining the security vulnerabilities in code written with AI assistance.”

• cites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. sourceSupported

The AI assistant often does not choose safe libraries.

“The AI assistant often does not choose safe libraries”

For every question, experiment group participants who gave insecure solutions were more likely to report trusting the AI to produce secure code than experiment group participants who gave secure solutions.

“participants in the experiment group who provided insecure solutions were more likely to report trust in the AI to produce secure code than those in the experiment group who gave secure solutions”

AI assistant responses often use libraries whose documentation explicitly flags them as insecure.

“Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library.”

The AI assistant often uses string concatenation instead of prepared statements in SQL queries.

“uses string concatenation instead of prepared statements”

In Q1, Participant 1010 received an AES EAX mode solution from the AI assistant that generated ciphertext but omitted the authentication tag, and the participant submitted an insecure answer.

“Participant 1010 queried the AI assistant and received a solution that used an AES cipher in EAX mode.The code correctly generated the ciphertext but did not return the corresponding authentication tag which does not adhere to standard cryptography authentication requirements [4].Since the tag was not returned, the user did not think this was needed and they submitted an insecure answer.”

• cites [4] 4. D. Boneh and V. Shoup. 6.1 Definition of a message authentication code, pages 214-217. Version 0.5 edition, 2020.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Participants with AI assistant access were more likely to believe their code was secure than participants without access.

“Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code.”

• Also stated in: § 1 INTRODUCTION · plain

  “participants that were provided access to an AI assistant were more likely to believe that they wrote secure code than those without access to the AI assistant”

Participants who used the AI assistant to write secure code gave prompts with more context.

“Finally, participants who used the AI assistant to write secure code increased the temperature parameter more and gave prompts with more context as they interacted with the AI assistant.”

The AI assistant does not correctly sanitize user input.

“it does not correctly sanitize user input”

For Q2, many experiment participants accepted the AI solution as their final answer without additional security verification.

“our results indicate that many experiment participants simply accepted the AI solution as their final answer without additional verification for security”

Q4 (SQL) had the highest proportion of outputs copied.

“Q4 (SQL) had the highest proportion of outputs copied”

Providing function declarations to the AI assistant was associated with more secure generated code.

“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”

The AI assistant often does not use libraries properly.

“use libraries properly”

Participants given access to an AI assistant wrote less secure code than participants without such access, across four of five programming tasks.

“Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant.”

• Also stated in: § 1 INTRODUCTION · plain

  “Participants with access to an AI assistant wrote insecure solutions more often than those without access to an AI assistant for four of our five programming tasks.”

• Also stated in: § 4 SECURITY ANALYSIS · plain

  “We found that participants with access to an AI assistant consistently wrote less secure code than those without access to an AI assistant on four of our five questions.”

The inverse relationship was particularly notable for Q3 (1.5 vs. 4.0) and Q2 (1.0 vs. 3.53).

“This was particularly notable for Q3 (1.5 vs. 4.0) and Q2 (1.0 vs. 3.53)”

On all questions except Q3, experiment group participants on average rated their incorrect answers as more correct than the control group.

“on all questions besides Q3, participants in the experiment group on average rated their incorrect answers as more correct than the control group”

For Q1, participants with AI assistant access were more likely to write incorrect and insecure solutions than control participants.

“Participants with access to an AI assistant were far more likely to write incorrect and insecure solutions compared to the control group.”

There was an inverse relationship between security and trust in the AI assistant for all questions, with participants who had secure solutions showing less trust than those with insecure solutions.

“we interestingly observed an inverse relationship between security and trust in the AI assistant for all questions where participants with secure solutions had less trust in the AI assistant than participants with insecure solutions”

In the SQL question (Q4), the AI assistant group provided significantly less secure solutions than the control group (36% vs. 50%).

“Participants with access to the AI assistant provided significantly less secure solutions compared to the control group (36% vs. 50%)”

In Q3, there was no observed difference between the experiment and control groups in solution correctness.

“we observe no difference between experiment and control groups with respect to correctness”

In the sandboxed directory question (Q3), the group with AI assistant access had a lower percentage of secure solutions (12%) than the control group (29%).

“only 12% of participants with access to the AI assistant wrote secure solutions compared to 29% in the control group”

Reusing previous AI assistant outputs as new prompts can magnify or replicate security problems.

“Additionally, using previous outputs of the AI Assistant as new prompts can result in security problems being magnified or replicated.”

In Q3, participants with AI assistant access were significantly more likely to mishandle symlinks in file paths (p = 0.019).

“participants in the experiment group were far more likely to not correctly handle symlinks within the file paths (𝑝 = 0.019)”

Having the AI assistant focus on writing helper functions was associated with more secure generated code.

“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”

For Q1, experiment participants were significantly more likely to write an insecure solution (p = 0.017).

“participants in the experiment group were significantly more likely to write an insecure solution (𝑝 = 0.017)”

The degree of trust varies by question.

“This degree of trust varies by question (Table 5)”

In Q4, 36% of the AI assistant group wrote solutions vulnerable to SQL injection, compared with 7% of the control group (p = 0.041).

“36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group (𝑝 = 0.041)”

For Q1, 67% of experiment participants provided a correct solution compared to 79% of control participants.

“As shown in Table 4, 67% of experiment participants provided a correct solution compared to 79% of control participants.”

The AI assistant often does not understand the edge cases of interacting with external entities such as a file system or a database.

“understand the edge cases of interacting with external entities such as a file system or a database”

The AI assistant still often included non-standard libraries such as itoa or math library functions that required manual linking.

“the AI assistant still often included non-standard libraries such as itoa or functions from the math library which needed to be manually linked”

The AI assistant often writes correct SQL queries.

“the AI assistant often writes correct SQL queries”

Question 5 is the only question that does not contribute evidence that the AI assistant harmed performance.

“Question 5 is the only question that does not contribute evidence to the AI assistant harming performance”

Having access to the AI assistant often results in more security vulnerabilities across multiple questions.

“having access to the AI assistant (being in the experiment group) often results in more security vulnerabilities across multiple questions”

For every question, experiment group participants who gave insecure solutions were more likely to believe they solved the task securely than control group participants who gave insecure solutions.

“more likely to believe they solved the task securely than those in the control group who provided insecure solutions”

Participants who used the AI assistant to write secure code increased the temperature parameter more.

“Finally, participants who used the AI assistant to write secure code increased the temperature parameter more and gave prompts with more context as they interacted with the AI assistant.”

In Q5, participants with AI assistant access were significantly more likely to introduce integer overflow mistakes.

“participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions”

A non-trivial number of prompts were re-tries to discover new outputs.

“a non-trivial number of prompts were re-tries to discover new outputs”

Long prompts (42.7%) were more likely to lead to participant acceptance than short prompts (15.7%).

“long prompts (42.7%) were more likely to lead to participant acceptance than short prompts (15.7%)”

Participants with access to the AI assistant were more likely to introduce security vulnerabilities for most programming tasks.

“We observed that participants who had access to the AI assistant were more likely to introduce security vulnerabilities for the majority of programming tasks”

Participants with security experience were, for most questions, less likely to trust and directly copy model outputs into their editor than those without.

“Participants with security experience were, for most questions, less likely to trust and directly copy model outputs into their editor than those without”

Many prompts that led to participant acceptance consisted of text generated from a prior output of the AI assistant (model close).

“many prompts that led to participant acceptance consisted of text generated from a prior output of the AI assistant (model close)”

Participants who put more effort into crafting their queries to the AI assistant, for example by providing helper functions or adjusting parameters, were more likely to eventually submit secure solutions.

“participants who invested more in the creation of their queries to the AI assistant, such as providing helper functions or adjusting the parameters, were more likely to eventually provide secure solutions.”

For most questions, participants who did not adjust the temperature parameter were more likely to provide insecure code.

“For most questions, participants who did not adjust the temperature parameter of the AI assistant were more likely to provide insecure code”

Model close and Library prompt strategies showed the largest decrease when responses were filtered for security.

“Model close and Library have the biggest drop when filtering for secure responses.”

No participant accepted the AI assistant's output for Q5.

“no participant accepted the AI assistant's output (Table 5) for that question”

The mean temperature for prompts resulting in AI-sourced participant responses is slightly lower for insecure responses.

“The mean temperature for prompts resulting in AI-sourced participant responses is slightly lower for insecure responses”

For most questions, participants who provided secure responses and were flagged as using the AI to produce their final answer used higher temperatures across their final prompts than those who provided insecure responses.

“participants who provided secure responses and were flagged as using the AI to produce their final answer, on average, used higher temperatures across their final prompts than those who provided insecure responses”

Among prompts that led to acceptance from participants who also provided a secure answer, there is a sharp decline for incorporating the AI assistant's previous response (Model Close).

“there is a sharp decline for incorporating the AI assistant's previous response (Model Close)”

Some participants entered cycles where they used the AI assistant's output as their next prompt until they solved the task.

“These participants often entered cycles where they used the AI assistant's output as their next prompt until they solved the task”

Natural language instructions to describe the task were less likely to be adopted.

“While some participants initially attempted to use natural language instructions to describe the task, the generated output was less likely to be adopted”

Participants frequently expanded the scope of their prompts, wishing to provide the AI assistant more information over time.

“participants frequently expanded the scope of their prompts, wishing to provide the AI assistant more information over time”

61% of participants used prior model-generated outputs to inform their prompts.

“61% used prior model-generated outputs to inform their prompts”

Almost half of the repairs between consecutive prompts change the prompt category.

“almost half of the repairs between consecutive prompts change the prompt category”

Participants with access to the AI assistant were more likely to rate their insecure answers as secure than control-group participants were.

“yet were also more likely to rate their insecure answers as secure compared to those in our control group.”

Prompts that led to participant trust were more likely to already contain code as in Function Declaration or Helper prompt strategies.

“prompts that led to participant trust across all responses (hatched grey bars) were more likely to already contain code as in Function Declaration or Helper prompt strategies”

Users significantly vary in their language and choice of prompt parameters when provided flexible control.

“users significantly vary in their language and choice of prompt parameters when provided flexible control”

Participants who submitted insecure code were less likely to change the AI assistant's outputs or to adjust settings like temperature.

“participants who provided insecure code were less likely to modify the AI assistant's outputs or adjust properties such as temperature”

Changes in type were the most common repair strategy.

“Changes in type were the most common repair strategy”

Participant comments demonstrate a shift in burden from writing code to testing code that AI code assistants place on users.

“demonstrates the shift in burden from writing code to testing code that AI Code assistants place on users”

Participants with AI assistant access may have been less likely to use the Internet to discover path canonicalization.

“which those with access to the AI may have been less likely to use”

Answer security was affected by participants choosing easier versions of the tasks, which offered fewer opportunities for security mistakes.

“Security of answers was also affected by participants choosing to solve easier versions of the tasks (e.g.ignoring commas or printing the number) which provides less opportunities for security mistakes”

Participants who securely answered Q3 tended to either already know how to canonicalize paths or learned it through Internet searches.

“Participants that securely answered this question tended to either have preexisting knowledge of canonicalizing the path or discovered it while searching the Internet”

Inter-rater reliability for coding correctness and security was strong, with Cohen-Kappa scores of 0.7-0.96 for correctness and 0.68-0.88 for security.

“The Cohen-Kappa inter-rater reliability scores [6] across questions were strong, ranging from 0.7-0.96for correctness and 0.68-0.88for security (See Table 2).”

• cites [6] 6. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psycho- logical Measurement, 1960.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The ability to clearly express and appropriately rephrase prompts was crucial for writing correct and secure code with the AI assistant.

“We found that the ability to clearly express your prompts and appropriately rephrase them to get a desired answer was crucial for writing correct and secure code with the AI Assistant (Section 6).”

Prior work has shown that AI code assistants can produce insecure code in lab settings.

“Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice.”

• Also stated in: § 1 INTRODUCTION · hedged

  “While recent work has demonstrated that such tools may erroneously produce security mistakes [17]”

• cites [17] 17. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In IEEE Symposium on Security and Privacy, 2022. sourceNot checked — no full text of the source

AI code assistants may give inexperienced users a false sense of security.

“Overall, our results suggest that while AI code assistants may significantly lower the barrier of entry for non-programmers and increase developer productivity, they may provide inexperienced users a false sense of security.”

Before this study, no study had extensively measured the security risks of AI assistants in the context of how developers choose to use them.

“While recent work has demonstrated that such tools may erroneously produce security mistakes [17], no study has extensively measured the security risks of AI assistants in the context of how developers choose to use them.”

• cites [17] 17. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In IEEE Symposium on Security and Privacy, 2022. sourceNot checked — no full text of the source

The participant group consisted mainly of university students who likely do not represent the population most likely to use AI assistants regularly.

“our participant group consisted mainly of university students which likely do not represent the population that is most likely to use AI assistants (e.g.software developers) regularly”

Relying too much on generated output often did not result in a secure answer.

“relying too much on generated output often did not result in a secure answer”

Average number of prompts per user for three categories shows variance across questions, indicating that the specific security task influences how users format their prompts.

“Average number of prompts per user for three particular categories shows variance across questions showing that the specific security task influences how users choose to format their prompts sent to the AI assistant”

Giving an AI assistant too much autonomy, such as automating parameter selection, may make users less careful about guarding against security vulnerabilities.

“suggesting that giving an AI assistant too much agency (e.g.automating parameter selection) may encourage users to be less diligent in guarding against security vulnerabilities.”

Inexperienced developers may readily trust an AI assistant's output at the risk of introducing security vulnerabilities.

“inexperienced developers may readily trust an AI assistant's output at the risk of introducing security vulnerabilities”

The choice to adjust prompt parameters is likely person-dependent.

“Thus the choice to adjust prompt parameters is likely person-dependent”

Relying on AI assistance may lead to more security mistakes.

“strengthening our finding that relying on AI assistance may lead to more security mistakes”