The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Do Users Write More Insecure Code with AI Assistants?
Open accessAbstract
AI code assistants have emerged as powerful tools that can aid in the software development life-cycle and can improve developer productivity. Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice. In this paper, we conduct a user study to examine how users interact with AI code assistants to solve a variety of security related tasks. Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant. Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code. To better inform the design of future AI-based code assistants, we release our user-study apparatus to researchers seeking to build on our work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
81 Own result · 18 Own method · 17 Own interpretation · 4 No full text of the sources · 3 Supported · 3 Sources not found · 1 Supported so far · 1 No basis given
Every source the paper cites for this claim was checked and supports it.
In the paper
“To the best of our knowledge, [21] is the only work that conducts a controlled user study examining the security vulnerabilities in code written with AI assistance.”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
The Introduction (p. 1) describes a controlled comparison: participants were randomly assigned to AI-assisted and unassisted groups, and their code was examined for security bugs. Section 5.1 (p. 13) says that, to the authors’ knowledge, this was the first user study measuring how LLM suggestions affect code security. That supports the citing sentence’s qualified identification of this work as the only prior study of its kind. (LLM, conf=0.90)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AI assistant often does not choose safe libraries”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment group who provided insecure solutions were more likely to report trust in the AI to produce secure code than those in the experiment group who gave secure solutions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“uses string concatenation instead of prepared statements”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Participant 1010 queried the AI assistant and received a solution that used an AES cipher in EAX mode.The code correctly generated the ciphertext but did not return the corresponding authentication tag which does not adhere to standard cryptography authentication requirements [4].Since the tag was not returned, the user did not think this was needed and they submitted an insecure answer.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
What the sources say
- Not matchedcites [4] 4. D. Boneh and V. Shoup. 6.1 Definition of a message authentication code, pages 214-217. Version 0.5 edition, 2020.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code.”
- Also stated in: § 1 INTRODUCTION · plain
“participants that were provided access to an AI assistant were more likely to believe that they wrote secure code than those without access to the AI assistant”
§ Do Users Write More Insecure Code with AI Assistants? > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, participants who used the AI assistant to write secure code increased the temperature parameter more and gave prompts with more context as they interacted with the AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it does not correctly sanitize user input”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our results indicate that many experiment participants simply accepted the AI solution as their final answer without additional verification for security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Q4 (SQL) had the highest proportion of outputs copied”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“use libraries properly”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant.”
- Also stated in: § 1 INTRODUCTION · plain
“Participants with access to an AI assistant wrote insecure solutions more often than those without access to an AI assistant for four of our five programming tasks.”
- Also stated in: § 4 SECURITY ANALYSIS · plain
“We found that participants with access to an AI assistant consistently wrote less secure code than those without access to an AI assistant on four of our five questions.”
§ Do Users Write More Insecure Code with AI Assistants? > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was particularly notable for Q3 (1.5 vs. 4.0) and Q2 (1.0 vs. 3.53)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on all questions besides Q3, participants in the experiment group on average rated their incorrect answers as more correct than the control group”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants with access to an AI assistant were far more likely to write incorrect and insecure solutions compared to the control group.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we interestingly observed an inverse relationship between security and trust in the AI assistant for all questions where participants with secure solutions had less trust in the AI assistant than participants with insecure solutions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants with access to the AI assistant provided significantly less secure solutions compared to the control group (36% vs. 50%)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we observe no difference between experiment and control groups with respect to correctness”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“only 12% of participants with access to the AI assistant wrote secure solutions compared to 29% in the control group”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, using previous outputs of the AI Assistant as new prompts can result in security problems being magnified or replicated.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment group were far more likely to not correctly handle symlinks within the file paths (𝑝 = 0.019)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment group were significantly more likely to write an insecure solution (𝑝 = 0.017)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This degree of trust varies by question (Table 5)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group (𝑝 = 0.041)”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 4, 67% of experiment participants provided a correct solution compared to 79% of control participants.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“understand the edge cases of interacting with external entities such as a file system or a database”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the AI assistant still often included non-standard libraries such as itoa or functions from the math library which needed to be manually linked”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the AI assistant often writes correct SQL queries”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Question 5 is the only question that does not contribute evidence to the AI assistant harming performance”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“having access to the AI assistant (being in the experiment group) often results in more security vulnerabilities across multiple questions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“more likely to believe they solved the task securely than those in the control group who provided insecure solutions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, participants who used the AI assistant to write secure code increased the temperature parameter more and gave prompts with more context as they interacted with the AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a non-trivial number of prompts were re-tries to discover new outputs”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“long prompts (42.7%) were more likely to lead to participant acceptance than short prompts (15.7%)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that participants who had access to the AI assistant were more likely to introduce security vulnerabilities for the majority of programming tasks”
§ Do Users Write More Insecure Code with AI Assistants? > 8 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants with security experience were, for most questions, less likely to trust and directly copy model outputs into their editor than those without”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“many prompts that led to participant acceptance consisted of text generated from a prior output of the AI assistant (model close)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants who invested more in the creation of their queries to the AI assistant, such as providing helper functions or adjusting the parameters, were more likely to eventually provide secure solutions.”
§ Do Users Write More Insecure Code with AI Assistants? > 8 CONCLUSION
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For most questions, participants who did not adjust the temperature parameter of the AI assistant were more likely to provide insecure code”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Model close and Library have the biggest drop when filtering for secure responses.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“no participant accepted the AI assistant's output (Table 5) for that question”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean temperature for prompts resulting in AI-sourced participant responses is slightly lower for insecure responses”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants who provided secure responses and were flagged as using the AI to produce their final answer, on average, used higher temperatures across their final prompts than those who provided insecure responses”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is a sharp decline for incorporating the AI assistant's previous response (Model Close)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These participants often entered cycles where they used the AI assistant's output as their next prompt until they solved the task”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While some participants initially attempted to use natural language instructions to describe the task, the generated output was less likely to be adopted”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants frequently expanded the scope of their prompts, wishing to provide the AI assistant more information over time”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“61% used prior model-generated outputs to inform their prompts”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“almost half of the repairs between consecutive prompts change the prompt category”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“yet were also more likely to rate their insecure answers as secure compared to those in our control group.”
§ Do Users Write More Insecure Code with AI Assistants? > 8 CONCLUSION
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“prompts that led to participant trust across all responses (hatched grey bars) were more likely to already contain code as in Function Declaration or Helper prompt strategies”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“users significantly vary in their language and choice of prompt parameters when provided flexible control”
§ Do Users Write More Insecure Code with AI Assistants? > 7.3 Recommendations
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants who provided insecure code were less likely to modify the AI assistant's outputs or adjust properties such as temperature”
§ Do Users Write More Insecure Code with AI Assistants? > 7.3 Recommendations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Changes in type were the most common repair strategy”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“demonstrates the shift in burden from writing code to testing code that AI Code assistants place on users”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which those with access to the AI may have been less likely to use”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Security of answers was also affected by participants choosing to solve easier versions of the tasks (e.g.ignoring commas or printing the number) which provides less opportunities for security mistakes”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants that securely answered this question tended to either have preexisting knowledge of canonicalizing the path or discovered it while searching the Internet”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The Cohen-Kappa inter-rater reliability scores [6] across questions were strong, ranging from 0.7-0.96for correctness and 0.68-0.88for security (See Table 2).”
§ Do Users Write More Insecure Code with AI Assistants? > 3.4 Analysis Procedure
What the sources say
- Not matchedcites [6] 6. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psycho- logical Measurement, 1960.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the ability to clearly express your prompts and appropriately rephrase them to get a desired answer was crucial for writing correct and secure code with the AI Assistant (Section 6).”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice.”
- Also stated in: § 1 INTRODUCTION · hedged
“While recent work has demonstrated that such tools may erroneously produce security mistakes [17]”
§ Do Users Write More Insecure Code with AI Assistants? > Abstract
What the sources say
- Not checked — no full text of the sourcecites [17] 17. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In IEEE Symposium on Security and Privacy, 2022. source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, our results suggest that while AI code assistants may significantly lower the barrier of entry for non-programmers and increase developer productivity, they may provide inexperienced users a false sense of security.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“While recent work has demonstrated that such tools may erroneously produce security mistakes [17], no study has extensively measured the security risks of AI assistants in the context of how developers choose to use them.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [17] 17. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In IEEE Symposium on Security and Privacy, 2022. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“our participant group consisted mainly of university students which likely do not represent the population that is most likely to use AI assistants (e.g.software developers) regularly”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“relying too much on generated output often did not result in a secure answer”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Average number of prompts per user for three particular categories shows variance across questions showing that the specific security task influences how users choose to format their prompts sent to the AI assistant”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that giving an AI assistant too much agency (e.g.automating parameter selection) may encourage users to be less diligent in guarding against security vulnerabilities.”
§ Do Users Write More Insecure Code with AI Assistants? > 7.3 Recommendations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“inexperienced developers may readily trust an AI assistant's output at the risk of introducing security vulnerabilities”
§ Do Users Write More Insecure Code with AI Assistants? > 7 DISCUSSION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus the choice to adjust prompt parameters is likely person-dependent”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“strengthening our finding that relying on AI assistance may lead to more security mistakes”
§ Do Users Write More Insecure Code with AI Assistants? > 7.1 Degree of AI Influence on Responses
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 26
- The quote only partly states this claim. 5
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We chose our model by using the BIC [23] across the aggregated questions to select a single model after removing variables with high colinearity such as years of programming experience, highest level of education completed, and current degree program.We then added variables that we explicitly wanted to control for, such as student status and years of programming experience.We correct for multiple”
§ Do Users Write More Insecure Code with AI Assistants? > 4 SECURITY ANALYSIS
What the sources say
- Supportedcites [23] 23. G. Schwarz. Estimating the Dimension of a Model. The Annals of Statistics, 1978. source
The citation is attached to BIC, not to the citing authors’ decisions about collinearity, controls, or multiple-testing correction. In the Introduction (p. 1), Schwarz gives a criterion for choosing among models by maximizing log likelihood minus a penalty based on model dimension and sample size. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [3] 3. Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 1995. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“[21] finds similar results and only examines a low level question in C”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
The Introduction (p. 2) describes the study as examining a low-level linked-list implementation in C and reports minimal security impact from AI assistance. Section 3.3 (p. 5) confirms that participants completed one C programming assignment, comprising 11 functions; the paper does not identify it as a single isolated question. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“While [21] finds inconclusive results with respect to the effect of AI Assistance on the degree of security vulnerabilities, we only observe mixed effects on our C task-emphasizing the importance of studying tasks across multiple languages.”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
Section 5.1 (p. 13) says the study found no conclusive evidence that AI assistance increases CWE incidence, including severe CWEs. That supports the citing sentence’s characterization of this source’s results as inconclusive on security vulnerabilities. The “mixed effects on our C task” are the citing authors’ own finding, not a claim attributed to this source. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common error made by participants in the experiment group was due to unsafe randomness which is a direct result of the library used to perform the task.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“many participants struggled with getting the AI assistant to output C code as the AI assistant often provided Javascript code (from comments using //) or Go code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant 1008 queried the AI assistant and received a correct but insecure solution that was missing an import statement.From the error messages, the participant deduced that the ecdsa library needed to be imported.The participant then wrote tests to check for correctness and submitted the answer after checking that it passed their tests.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment on average believed their answers were more secure than those in the control group despite often providing more insecure answers”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while 64% of participants in the control group provided correct solutions, only 52% of participants in the experiment group did so”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we find evidence that participants do adjust model parameters for different tasks when given the opportunity (influencing correctness and security of their responses).”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“also significantly more likely to use trivial ciphers such as substitution ciphers (𝑝 = 0.018).”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A combination of adjusting temperature, instructing the AI assistant to use C via comments, and writing function headers lead to more successful C queries”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants with access to the AI assistant wrote more partially correct code but less correct and incorrect code than the control group and with no large differences in security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly to Q1, we observe that participants who had access to the AI produced less secure answers compared to those in the control group with only 3% of participants with access to an AI writing a secure solution compared to 21% of the control group (𝑝 = 0.039).”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Participants that securely answered this question used deterministic ECDSA signatures (RFC 6979 [19]) or used a library that the AI assistant was unlikely to choose-potentially due to bias in the training corpus available to the AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
What the sources say
- Not matchedcites RFC 6979 [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They were also less likely to authenticate the final returned value.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“several participants adding code such as helper functions as well as language about securityas shown in Figure 6”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“87% of secure responses required significant edits from users while partially secure and insecure responses varied broadly in terms of edit distance”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“49% specified the programming language”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: codex-davinci-002 is language-agnostic.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21% of users chose to provide the AI assistant with instructions”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Providing a function declaration is more common for Python questions (Q1, Q2); whereas, participants were more likely to specify the programming language for the SQL and C questions (Q4, Q5)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“64% of participants tried direct task specification”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On every type of security mistake across all five questions, the AI assistant was involved in at least as many mistakes as a participant and often the majority of mistakes”
§ Do Users Write More Insecure Code with AI Assistants? > 7.1 Degree of AI Influence on Responses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for Q5 (C), the AI output was never directly usedin part due to the difficulty of getting the AI assistant to return C code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“48.5% of participants never adjusted the temperature for any question and 51.5% never adjusted the response length”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“53% specified a particular library”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants in the experiment group made on average 4.6 queries to the AI assistant per question”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants chose to prompt the AI assistant with a variety of strategies (Table 6)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants adjusted the temperature values of their prompts with the mean number of unique temperature values across all prompts for a single question ranging from 1.21 (Q4) to 1.47 (Q5)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“adjusting response length had a mixed effect as this parameter only affects the amount of code generated”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we observe that the AI assistant often outputs code that, while satisfying “correctness”, has little context of what security properties a cipher should have, and in some cases, can create code that unintentionally confuses the user.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“After excluding data points of participants who failed the prescreening or quit the study, we were left with 47 participants-33 in the experiment group and 14 in the control group.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plainn = 47- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, two authors watched all of the screen recordings, noting the steps the participant followed to reach their answer and which mistakes resulted from these steps.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.4 Analysis Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In order to verify that participants had programming knowledge, we asked a brief prescreening question before proceeding with the study that focused on participants' ability to read and interpret a for-loop-which has been used in other user studies [7].”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
What the sources say
- Not checked — no full text of the sourcecites [7] 7. A. Danilova, A. Naiakshina, and M. Smith. One size does not fit all: A grounded theory and online survey study of developer preferences for security warning types. In IEEE/ACM 42nd International Conference on Software Engineering, 2020. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Ultimately, we recruited 54 participants ranging from early undergraduate students to industry professionals with decades of programming experience.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plainn = 54- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This does not pose any problems to our analysis due to the fact that all statistical tests conducted are valid for unequal sample sizes and variances (Welch's t-test and the Chi-squared test for categorical data).”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“At the beginning of the study, participants were randomly assigned to one of two groups-a control group, which was required to solve the programming questions without an AI assistant, and an experiment group, which was provided access to an AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“after the study we found that this question was too vague, as many participants simply called alert or console.log.We thus ignore this question in our analysis, focusing on the other five.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.1 Building Security-Related Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants were allowed to leave the study early and we did not observe fatigue playing a role in question answers.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.1 Building Security-Related Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we use multivariable regression to control for participants' security backgrounds when interpreting results in Section 4.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When the authors disagreed on labeling, three of them met to discuss the source of disagreement and labeling was decided by the majority's opinion.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.4 Analysis Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2 : Inter-rater Reliability Scores for Correctness and Security across all 5 questions.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that temperature variation may be to try to get the model to produce outputs participants wish to accept”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The B-H crit column contains the critical values needed for statistical significance after the Benjamini-Hochberg correction.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“this metric fails to account for situations where the AI's output may influence a user's response without being copied directly, as well as edits a user may perform on the generated output in order to improve its correctness or security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Security was captured in the definition, making it difficult to disentangle correctness and security.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the artificial environment of our study-such as time constraints and participants' performance not impacting their jobs-does not perfectly capture real working conditions and restricts how results generalize to real conditions”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likely due to a combination of user unfamiliarity with Javascript and the AI assistant's ability to generate Javascript code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“This supports [12]'s findings on the implications of developers' syntax on an AI assistant for web applications”
§ Do Users Write More Insecure Code with AI Assistants? > 7.3 Recommendations
What the sources say
- Not checked — no full text of the sourcecites [12] 12. E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry. Discovering the syntax and strategies of natural language programming with generative language models. In ACM CHI Conference on Human Factors in Computing Systems, 2022. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it can write correct answers (and we observe no difference between experiment and control groups with respect to correctness) but often fails to cover edge cases”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI assistant (codex-davinci-002) lacks information about the structure of file paths or symlinks.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“it is challenging to ascertain the degree the AI assistant actually influenced a participant's response”
§ Do Users Write More Insecure Code with AI Assistants? > 7.1 Degree of AI Influence on Responses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which are unlikely to appear in GitHub source code and out-of-domain of codex-davinci-002's underlying training data”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“a larger sample size would be necessary to assess more subtle effects”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“aspects such as the location of the AI assistant or the latency in making query requests may have affected our overall results”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
The AI assistant used in the experiment group is representative of AI code assistants generally.
“Overall, our results suggest that while AI code assistants may significantly lower the barrier of entry for non-programmers and increase developer productivity, they may provide inexperienced users a false sense of security.”
The paper relies on this without support.
The security-related programming tasks used in the study are representative of real-world security-related programming tasks.
“We chose questions that were self contained, could be solved in a short amount of time, and covered a wide breadth of potential security mistakes that are commonly taught in introductory computer security courses.”
The paper relies on this without support.
The AI assistant (codex-davinci-002) lacks information about the structure of file paths or symlinks.
“Since the AI assistant does not have information about the structure of file paths or symlinks”
The paper relies on this without support.
codex-davinci-002 is language-agnostic.
“as codex-davinci-002 itself is language-agnostic”
The paper relies on this without support.
The study is the first user study to examine how people interact with an AI code assistant to solve security-related tasks across multiple programming languages.
“We conducted the first user study examining how people interact with an AI code assistant (built with OpenAI's Codex) to solve a variety of security related tasks across different programming languages.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“previous research found that developers who work with AI assistants tend to write less secure code (Perry et al. 2023).”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Perry et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“participants who had access to an AI assistant tended to write less secure code compared to those without [32]”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Perry et al. [32] observed that AI-assisted developers introduced more vulnerabilities.”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Thus, our more experienced participants perhaps correctly perceive their proficiency in secure coding to degrade when using GAI tools.”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Automated code generation with large language models has gained significant traction, but there remains no guarantee on the correctness of generated code.”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Perry et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 4 SECURITY ANALYSIS
“We note statistically significant differences between experiment and control groups in the text for each task using a Chi-squared unequal variance test for categorical variables.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
“we determine statistical significance of this treatment for particular security buckets (e.g.only "Insecure") using Welch's unequal variance t-test in our main reported results.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
“static analysis tools (e.g.Spot-Bugs [24] and Infer [8])”
§ Do Users Write More Insecure Code with AI Assistants? > References
“24. M. Tabachnyk and S. Nikolov. Ml-enhanced code completion improves developer productivity. https://ai.googleblog.com/2022/07/ml-enhanced-code-completion- improves.html, Jul 2022.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 9.5 Demographics
“Table 7 contains more detailed demographics on the participant population for the experiment and control groups.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
“Table: 7 Table 7 : Proportion of prompts and users for repair strategies across all questions.”
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038
Journal of the Royal Statistical Society: Series B (Methodological)
(2018) · Journal of the Royal Statistical Society Series B (Methodological) · doi:10.1111/(issn)2517-6161 · Cited by 1,612
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Birch, A., & Osborne, M. (2011) · The Association for Computational Linguistics · Cited by 1,196
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Research works printed but not linked: 3
- D. Boneh and V. Shoup. 6.1 Definition of a message authentication code, pages 214--217. Version 0.5 edition, 2020.
- G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022.
- M. Tabachnyk and S. Nikolov. Ml-enhanced code completion improves developer productivity. https://ai.googleblog.com/2022/07/ml-enhanced-code-completionimproves.html, Jul 2022.
Cited by
Sorted by citations
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
The impact of ChatGPT on higher education
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1206936 · Cited by 519
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Perry et al. recently examined whether novices using AI code generation tools wrote more secure code, finding that novices consistently wrote insecure code with specific high vulnerabilities in string encryption and SQL injection [29].”
Background
Metadata
- Authors
- Perry, N., Srivastava, M., Kumar, D., & Boneh, D.
- Affiliations
- Stanford University (US)
- University of California San Diego (US)
- Venue
- Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2023-11-15
- Pages
- 2785-2799
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W4388858772
- Cited by
- 259
- References
- 23
- Citations by year
- 2026: 99 · 2025: 83 · 2024: 55 · 2023: 22
- FWCI
- 66.42
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1656518)
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- software security, security vulnerabilities, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
