The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Do Users Write More Insecure Code with AI Assistants?
Open accessAbstract
AI code assistants have emerged as powerful tools that can aid in the software development life-cycle and can improve developer productivity. Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice. In this paper, we conduct a user study to examine how users interact with AI code assistants to solve a variety of security related tasks. Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant. Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code. To better inform the design of future AI-based code assistants, we release our user-study apparatus to researchers seeking to build on our work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
81 Own result · 18 Own method · 17 Own interpretation · 4 No full text of the sources · 3 Supported · 3 Sources not found · 1 Supported so far · 1 No basis given
Every source the paper cites for this claim was checked and supports it.
In the paper
“To the best of our knowledge, [21] is the only work that conducts a controlled user study examining the security vulnerabilities in code written with AI assistance.”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
The Introduction (p. 1) describes a controlled comparison: participants were randomly assigned to AI-assisted and unassisted groups, and their code was examined for security bugs. Section 5.1 (p. 13) says that, to the authors’ knowledge, this was the first user study measuring how LLM suggestions affect code security. That supports the citing sentence’s qualified identification of this work as the only prior study of its kind. (LLM, conf=0.90)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AI assistant often does not choose safe libraries”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment group who provided insecure solutions were more likely to report trust in the AI to produce secure code than those in the experiment group who gave secure solutions”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“uses string concatenation instead of prepared statements”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Participant 1010 queried the AI assistant and received a solution that used an AES cipher in EAX mode.The code correctly generated the ciphertext but did not return the corresponding authentication tag which does not adhere to standard cryptography authentication requirements [4].Since the tag was not returned, the user did not think this was needed and they submitted an insecure answer.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
What the sources say
- Not matchedcites [4] 4. D. Boneh and V. Shoup. 6.1 Definition of a message authentication code, pages 214-217. Version 0.5 edition, 2020.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code.”
- Also stated in: § 1 INTRODUCTION · plain
“participants that were provided access to an AI assistant were more likely to believe that they wrote secure code than those without access to the AI assistant”
§ Do Users Write More Insecure Code with AI Assistants? > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, participants who used the AI assistant to write secure code increased the temperature parameter more and gave prompts with more context as they interacted with the AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it does not correctly sanitize user input”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our results indicate that many experiment participants simply accepted the AI solution as their final answer without additional verification for security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 26
- The quote only partly states this claim. 5
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We chose our model by using the BIC [23] across the aggregated questions to select a single model after removing variables with high colinearity such as years of programming experience, highest level of education completed, and current degree program.We then added variables that we explicitly wanted to control for, such as student status and years of programming experience.We correct for multiple”
§ Do Users Write More Insecure Code with AI Assistants? > 4 SECURITY ANALYSIS
What the sources say
- Supportedcites [23] 23. G. Schwarz. Estimating the Dimension of a Model. The Annals of Statistics, 1978. source
The citation is attached to BIC, not to the citing authors’ decisions about collinearity, controls, or multiple-testing correction. In the Introduction (p. 1), Schwarz gives a criterion for choosing among models by maximizing log likelihood minus a penalty based on model dimension and sample size. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [3] 3. Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 1995. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“[21] finds similar results and only examines a low level question in C”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
The Introduction (p. 2) describes the study as examining a low-level linked-list implementation in C and reports minimal security impact from AI assistance. Section 3.3 (p. 5) confirms that participants completed one C programming assignment, comprising 11 functions; the paper does not identify it as a single isolated question. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“While [21] finds inconclusive results with respect to the effect of AI Assistance on the degree of security vulnerabilities, we only observe mixed effects on our C task-emphasizing the importance of studying tasks across multiple languages.”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
What the sources say
- Supportedcites [21] 21. G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022. source
Section 5.1 (p. 13) says the study found no conclusive evidence that AI assistance increases CWE incidence, including severe CWEs. That supports the citing sentence’s characterization of this source’s results as inconclusive on security vulnerabilities. The “mixed effects on our C task” are the citing authors’ own finding, not a claim attributed to this source. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that those who specified task instructions, provided function declarations to use, and had the AI Assistant focus on writing helper functions generated more secure code.”
§ Do Users Write More Insecure Code with AI Assistants? > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common error made by participants in the experiment group was due to unsafe randomness which is a direct result of the library used to perform the task.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“many participants struggled with getting the AI assistant to output C code as the AI assistant often provided Javascript code (from comments using //) or Go code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant 1008 queried the AI assistant and received a correct but insecure solution that was missing an import statement.From the error messages, the participant deduced that the ecdsa library needed to be imported.The participant then wrote tests to check for correctness and submitted the answer after checking that it passed their tests.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in the experiment on average believed their answers were more secure than those in the control group despite often providing more insecure answers”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while 64% of participants in the control group provided correct solutions, only 52% of participants in the experiment group did so”
§ Do Users Write More Insecure Code with AI Assistants? > 4.4 Q4: SQL
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we find evidence that participants do adjust model parameters for different tasks when given the opportunity (influencing correctness and security of their responses).”
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“also significantly more likely to use trivial ciphers such as substitution ciphers (𝑝 = 0.018).”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A combination of adjusting temperature, instructing the AI assistant to use C via comments, and writing function headers lead to more successful C queries”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants with access to the AI assistant wrote more partially correct code but less correct and incorrect code than the control group and with no large differences in security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.5 Q5: C Strings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly to Q1, we observe that participants who had access to the AI produced less secure answers compared to those in the control group with only 3% of participants with access to an AI writing a secure solution compared to 21% of the control group (𝑝 = 0.039).”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Participants that securely answered this question used deterministic ECDSA signatures (RFC 6979 [19]) or used a library that the AI assistant was unlikely to choose-potentially due to bias in the training corpus available to the AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.2 Q2: Signing a Message
What the sources say
- Not matchedcites RFC 6979 [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They were also less likely to authenticate the final returned value.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“several participants adding code such as helper functions as well as language about securityas shown in Figure 6”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“87% of secure responses required significant edits from users while partially secure and insecure responses varied broadly in terms of edit distance”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“49% specified the programming language”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: codex-davinci-002 is language-agnostic.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“21% of users chose to provide the AI assistant with instructions”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Providing a function declaration is more common for Python questions (Q1, Q2); whereas, participants were more likely to specify the programming language for the SQL and C questions (Q4, Q5)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“64% of participants tried direct task specification”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On every type of security mistake across all five questions, the AI assistant was involved in at least as many mistakes as a participant and often the majority of mistakes”
§ Do Users Write More Insecure Code with AI Assistants? > 7.1 Degree of AI Influence on Responses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for Q5 (C), the AI output was never directly usedin part due to the difficulty of getting the AI assistant to return C code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“48.5% of participants never adjusted the temperature for any question and 51.5% never adjusted the response length”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“53% specified a particular library”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants in the experiment group made on average 4.6 queries to the AI assistant per question”
§ Do Users Write More Insecure Code with AI Assistants? > 6.3 Repair Strategies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants chose to prompt the AI assistant with a variety of strategies (Table 6)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants adjusted the temperature values of their prompts with the mean number of unique temperature values across all prompts for a single question ranging from 1.21 (Q4) to 1.47 (Q5)”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“adjusting response length had a mixed effect as this parameter only affects the amount of code generated”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we observe that the AI assistant often outputs code that, while satisfying “correctness”, has little context of what security properties a cipher should have, and in some cases, can create code that unintentionally confuses the user.”
§ Do Users Write More Insecure Code with AI Assistants? > 4.1 Q1: Encryption & Decryption
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“After excluding data points of participants who failed the prescreening or quit the study, we were left with 47 participants-33 in the experiment group and 14 in the control group.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plainn = 47- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, two authors watched all of the screen recordings, noting the steps the participant followed to reach their answer and which mistakes resulted from these steps.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.4 Analysis Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In order to verify that participants had programming knowledge, we asked a brief prescreening question before proceeding with the study that focused on participants' ability to read and interpret a for-loop-which has been used in other user studies [7].”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
What the sources say
- Not checked — no full text of the sourcecites [7] 7. A. Danilova, A. Naiakshina, and M. Smith. One size does not fit all: A grounded theory and online survey study of developer preferences for security warning types. In IEEE/ACM 42nd International Conference on Software Engineering, 2020. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Ultimately, we recruited 54 participants ranging from early undergraduate students to industry professionals with decades of programming experience.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plainn = 54- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This does not pose any problems to our analysis due to the fact that all statistical tests conducted are valid for unequal sample sizes and variances (Welch's t-test and the Chi-squared test for categorical data).”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“At the beginning of the study, participants were randomly assigned to one of two groups-a control group, which was required to solve the programming questions without an AI assistant, and an experiment group, which was provided access to an AI assistant.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“after the study we found that this question was too vague, as many participants simply called alert or console.log.We thus ignore this question in our analysis, focusing on the other five.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.1 Building Security-Related Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants were allowed to leave the study early and we did not observe fatigue playing a role in question answers.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.1 Building Security-Related Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we use multivariable regression to control for participants' security backgrounds when interpreting results in Section 4.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.2 Recruitment and Participant Pool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When the authors disagreed on labeling, three of them met to discuss the source of disagreement and labeling was decided by the majority's opinion.”
§ Do Users Write More Insecure Code with AI Assistants? > 3.4 Analysis Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2 : Inter-rater Reliability Scores for Correctness and Security across all 5 questions.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that temperature variation may be to try to get the model to produce outputs participants wish to accept”
§ Do Users Write More Insecure Code with AI Assistants? > 6.2 Prompt Parameters
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The B-H crit column contains the critical values needed for statistical significance after the Benjamini-Hochberg correction.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“this metric fails to account for situations where the AI's output may influence a user's response without being copied directly, as well as edits a user may perform on the generated output in order to improve its correctness or security”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Security was captured in the definition, making it difficult to disentangle correctness and security.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the artificial environment of our study-such as time constraints and participants' performance not impacting their jobs-does not perfectly capture real working conditions and restricts how results generalize to real conditions”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likely due to a combination of user unfamiliarity with Javascript and the AI assistant's ability to generate Javascript code”
§ Do Users Write More Insecure Code with AI Assistants? > 4.6 Security Results Summary
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“This supports [12]'s findings on the implications of developers' syntax on an AI assistant for web applications”
§ Do Users Write More Insecure Code with AI Assistants? > 7.3 Recommendations
What the sources say
- Not checked — no full text of the sourcecites [12] 12. E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry. Discovering the syntax and strategies of natural language programming with generative language models. In ACM CHI Conference on Human Factors in Computing Systems, 2022. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it can write correct answers (and we observe no difference between experiment and control groups with respect to correctness) but often fails to cover edge cases”
§ Do Users Write More Insecure Code with AI Assistants? > 4.3 Q3: Sandboxed Directory
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI assistant (codex-davinci-002) lacks information about the structure of file paths or symlinks.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“it is challenging to ascertain the degree the AI assistant actually influenced a participant's response”
§ Do Users Write More Insecure Code with AI Assistants? > 7.1 Degree of AI Influence on Responses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which are unlikely to appear in GitHub source code and out-of-domain of codex-davinci-002's underlying training data”
§ Do Users Write More Insecure Code with AI Assistants? > 6.1 Prompt Language
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“a larger sample size would be necessary to assess more subtle effects”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“aspects such as the location of the AI assistant or the latency in making query requests may have affected our overall results”
§ Do Users Write More Insecure Code with AI Assistants? > 7.2 Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
The AI assistant used in the experiment group is representative of AI code assistants generally.
“Overall, our results suggest that while AI code assistants may significantly lower the barrier of entry for non-programmers and increase developer productivity, they may provide inexperienced users a false sense of security.”
The paper relies on this without support.
The security-related programming tasks used in the study are representative of real-world security-related programming tasks.
“We chose questions that were self contained, could be solved in a short amount of time, and covered a wide breadth of potential security mistakes that are commonly taught in introductory computer security courses.”
The paper relies on this without support.
The AI assistant (codex-davinci-002) lacks information about the structure of file paths or symlinks.
“Since the AI assistant does not have information about the structure of file paths or symlinks”
The paper relies on this without support.
codex-davinci-002 is language-agnostic.
“as codex-davinci-002 itself is language-agnostic”
The paper relies on this without support.
The study is the first user study to examine how people interact with an AI code assistant to solve security-related tasks across multiple programming languages.
“We conducted the first user study examining how people interact with an AI code assistant (built with OpenAI's Codex) to solve a variety of security related tasks across different programming languages.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“previous research found that developers who work with AI assistants tend to write less secure code (Perry et al. 2023).”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Perry et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“participants who had access to an AI assistant tended to write less secure code compared to those without [32]”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Perry et al. [32] observed that AI-assisted developers introduced more vulnerabilities.”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Thus, our more experienced participants perhaps correctly perceive their proficiency in secure coding to degrade when using GAI tools.”
From My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“Automated code generation with large language models has gained significant traction, but there remains no guarantee on the correctness of generated code.”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Perry et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 4 SECURITY ANALYSIS
“We note statistically significant differences between experiment and control groups in the text for each task using a Chi-squared unequal variance test for categorical variables.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
“we determine statistical significance of this treatment for particular security buckets (e.g.only "Insecure") using Welch's unequal variance t-test in our main reported results.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 2 BACKGROUND & RELATED WORK
“static analysis tools (e.g.Spot-Bugs [24] and Infer [8])”
§ Do Users Write More Insecure Code with AI Assistants? > References
“24. M. Tabachnyk and S. Nikolov. Ml-enhanced code completion improves developer productivity. https://ai.googleblog.com/2022/07/ml-enhanced-code-completion- improves.html, Jul 2022.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do Users Write More Insecure Code with AI Assistants? > 9.5 Demographics
“Table 7 contains more detailed demographics on the participant population for the experiment and control groups.”
§ Do Users Write More Insecure Code with AI Assistants? > 9.6 Regression Tables
“Table: 7 Table 7 : Proportion of prompts and users for repair strategies across all questions.”
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038
Journal of the Royal Statistical Society: Series B (Methodological)
(2018) · Journal of the Royal Statistical Society Series B (Methodological) · doi:10.1111/(issn)2517-6161 · Cited by 1,612
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Birch, A., & Osborne, M. (2011) · The Association for Computational Linguistics · Cited by 1,196
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Research works printed but not linked: 3
- D. Boneh and V. Shoup. 6.1 Definition of a message authentication code, pages 214--217. Version 0.5 edition, 2020.
- G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg. Security implications of large language model code assistants: A user study. https://arxiv. org/abs/2208.09727, 2022.
- M. Tabachnyk and S. Nikolov. Ml-enhanced code completion improves developer productivity. https://ai.googleblog.com/2022/07/ml-enhanced-code-completionimproves.html, Jul 2022.
Cited by
Sorted by citations
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
The impact of ChatGPT on higher education
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1206936 · Cited by 519
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Perry et al. recently examined whether novices using AI code generation tools wrote more secure code, finding that novices consistently wrote insecure code with specific high vulnerabilities in string encryption and SQL injection [29].”
BackgroundThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“…meta-d ’ estimates How do aspects of GenAI output (e.g., the speed at which it is produced, its verbal fluency in the case of text) serve as heuristic cues that influence users’ confidence in the output and their evaluation ability, as well as the amount of effort they invest into evaluation?”
BackgroundÖzkaya, İ. (2023) · IEEE Software · doi:10.1109/ms.2023.3248401 · Cited by 265
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
Ullah, S., Han, M., Pujar, S., Pearce, H. A., Coskun, A. K., & Stringhini, G. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00210 · Cited by 208
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120
“In practice, this can often lead to users accepting code with subtle bugs while using LLMs [12], [13].”
BackgroundPrather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 168
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“Meanwhile, Perry et al. [2022] find that humans miss security vulnerabilities introduced by LLM code assistants.”
BackgroundInvestigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
“A related approach is to support the interpretability of model mechanisms [37, 43, 53], increasing the predictability [18, 21] of AI behavior.”
BackgroundMethodA Performance Study of LLM-Generated Code on Leetcode
Coignion, T., Quinton, C., & Rouvoy, R. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661221 · Cited by 88
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69
Democratising AI: Multiple Meanings, Goals, and Methods
Seger, E., Ovadya, A., Siddarth, D., Garfinkel, B., & Dafoe, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604693 · Cited by 75
Zhong, L., & Wang, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30185 · Cited by 96
Large Language Models and Simple, Stupid Bugs
Jesse, K., Ahmed, T., Dévanbu, P., & Morgan, E. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00082 · Cited by 83
The FormAI Dataset: Generative AI in Software Security through the Lens of Formal Verification
Tihanyi, N., Bisztray, T., Jain, R., Ferrag, M. A., Cordeiro, L. C., & Mavroeidis, V. (2023) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3617555.3617874 · Cited by 70
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716848 · Cited by 205
How are AI assistants changing higher education?
Schön, E.-M., Neumann, M., Hofmann-Stölting, C., Baeza-Yates, R. A., & Rauschenberger, M. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1208550 · Cited by 68
How secure is AI-generated code: a large-scale comparison of large language models
Tihanyi, N., Bisztray, T., Ferrag, M. A., Jain, R., & Cordeiro, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10590-1 · Cited by 95
A systematic literature review on the impact of AI models on the security of code generation
Negri-Ribalta, C., Géraud, R., Sergeeva, A., & Lenzini, G. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1386720 · Cited by 67
Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 66
Automatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 83
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models
Aghakhani, H., Dai, W., Manoel, A., Fernandes, X., Kharkar, A., Kruegel, C., Vigna, G., Evans, D., Zorn, B., & Sim, R. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00140 · Cited by 72
LLM-Powered Static Binary Taint Analysis
Liu, P., Sun, C. P., Zheng, Y., Feng, X., Qin, C., Wang, Y., Xu, Z., Li, Z., Di, P., Jiang, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711816 · Cited by 96
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43
Klemmer, J. H., Horstmann, S. A., Patnaik, N., Ludden, C., Burton, C., Powers, C., Massacci, F., Rahman, A., Votipka, D., Lipford, H. R., et al. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3690283 · Cited by 41
LLMs in Web Development: Evaluating LLM-Generated PHP Code Unveiling Vulnerabilities and Limitations
Tóth, R., Bisztray, T., & Erdődi, L. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-68738-9_34 · Cited by 34
Assessing the Security of GitHub Copilot's Generated Code - A Targeted Replication Study
Majdinasab, V., Bishop, M. J., Rasheed, S., Dakhel, A. M., Tahir, A., & Khomh, F. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00051 · Cited by 66
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Hamer, S., d’Amorim, M., & Williams, L. A. (2024) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00014 · Cited by 52
Nazzal, M., Khalil, I. M., Khreishah, A. A., & Phan, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690298 · Cited by 33
Training large language models on narrow tasks can lead to broad misalignment
Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., Srivastava, M., Labenz, N., & Evans, O. (2026) · Nature · doi:10.1038/s41586-025-09937-5 · Cited by 31
“It is well-known that present-day language models can exhibit a wide range of potentially harmful behaviour in response to benign queries from users, from generating insecure code to encouraging self-harm 29,30 .”
Self-citationValidating AI-Generated Code with Live Programming
Ferdowsi, K., Huang, R. L., James, M. B., Polikarpova, N., & Lerner, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642495 · Cited by 51
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27
“Blind trust in these systems may lead to potential negative impact, as there might be cases where the generated code is insecure (Perry et al., 2022) or contains sensitive information.”
BackgroundPrompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 85
Hasan, W. U., Zaman, K. T., Wang, X., Li, J., Xie, B., & Tao, C. (2024) · npj Biomedical Innovations. · doi:10.1038/s44385-024-00004-8 · Cited by 32
Oh, S., Lee, K., Park, S., Kim, D., & Kim, H. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00046 · Cited by 27
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
An Investigation into Misuse of Java Security APIs by Large Language Models
Mousavi, Z., Islam, C., Moore, K., Abuadbba, A., & Babar, M. A. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3661134 · Cited by 28
Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
Wang, W., Ning, H., Zhang, G., Liu, L., & Wang, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643758 · Cited by 40
Identifying the Factors That Influence Trust in AI Code Completion
Brown, A., D'Angelo, S., Murillo, A., Jaspan, C., & Green, C. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664757 · Cited by 20
From Today’s Code to Tomorrow’s Symphony: The AI Transformation of Developer’s Routine by 2030
Qiu, K., Puccinelli, N., Ciniselli, M., & Grazia, L. D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709353 · Cited by 44
Guiding Enumerative Program Synthesis with Large Language Models
Li, Y., Parsert, J., & Polgreen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65630-9_15 · Cited by 32
Students' Use of GitHub Copilot for Working with Large Code Bases
Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701800 · Cited by 19
Do LLMs consider security? an empirical study on responses to programming questions
Sajadi, A., Le, B., Nguyen, A., Damevski, K., & Chatterjee, P. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10658-6 · Cited by 27
Experiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 32
Generative AI Uses and Risks for Knowledge Workers in a Science Organization
Wagman, K. B., Dearing, M. T., & Chetty, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713827 · Cited by 23
Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models
Bruni, M., Gabrielli, F., Ghafari, M., et al. (2025) · 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge) · doi:10.1109/forge66646.2025.00018 · Cited by 43
A User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 31
Navigating Complexity in Software Engineering: A Prototype for Comparing GPT-n Solutions
Treude, C. (2023) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse59190.2023.00008 · Cited by 17
Mutation Testing in Practice: Insights From Open-Source Software Developers
Sánchez, A. B., Parejo, J. A., Segura, S., et al. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3377378 · Cited by 19
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 18
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 39
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
Cyber Shadows: Neutralizing Security Threats With AI and Targeted Policy Measures
Schmitt, M., & Koutroumpis, P. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3527398 · Cited by 24
Chatting with AI: Deciphering Developer Conversations with ChatGPT
Mohamed, S., Parvin, A., & Parra, E. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645078 · Cited by 12
PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
Peng, Y. F., Gotmare, A. D., Lyu, M. R.-T., Xiong, C., Savarese, S., & Sahoo, D. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00008 · Cited by 56
Assessing the Effectiveness and Security Implications of AI Code Generators
Taeb, M., Chi, H., & Bernadin, S. (2024) · Journal of The Colloquium for Information Systems Security Education · doi:10.53735/cisse.v11i1.180 · Cited by 13
A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama
Cursaru, V.-A., Duits, L., Milligan, J., Ural, D., Sanchez, B. R., Stoico, V., & Malavolta, I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70245-7_12 · Cited by 17
On Automated Assistants for Software Development: The Role of LLMs
Leung, M., & Murphy, G. C. (2023) · International Conference on Automated Software Engineering · doi:10.1109/ase56229.2023.00035 · Cited by 12
Perceived Impact of AI-Based Tooling on Software Development Code Quality
Martinović, B., & Rozić, R. (2025) · SN Computer Science · doi:10.1007/s42979-024-03608-4 · Cited by 12
ChatGPT vs. Stack Overflow: An Exploratory Comparison of Programming Assistance Tools
Liu, J., Tang, X., Li, L., Chen, P., & Liu, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c60940.2023.00105 · Cited by 15
Analysing the Role of Generative AI in Software Engineering - Results from an MLR
Bazzan, T., Olojo, B., Majda, P., Kelly, T., Yılmaz, M., Marks, G., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_11 · Cited by 16
Evaluating the Impact of ChatGPT on Exercises of a Software Security Course
Li, J., Meland, P. H., Notland, J. S., Storhaug, A., & Tysse, J. H. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304857 · Cited by 12
Software Security Analysis in 2030 and Beyond: A Research Roadmap
Böhme, M., Bodden, E., Bultan, T., Cadar, C., Liu, Y., & Scanniello, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708533 · Cited by 21
Ethical Issues in AI-Generated Texts: A Systematic Review and Analysis
Zhang, Y., Zhou, T., Qiao, H., & Li, T. X. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2530071 · Cited by 17
“Who will be left behind?”: A Swedish case of learning AI in vocational education
Metreveli, A., Chen, X., Hedman, A., & Sergeeva, A. (2025) · International Journal of Educational Research · doi:10.1016/j.ijer.2025.102697 · Cited by 17
Anagnostopoulos, C. (2023) · F1000Research · doi:10.12688/f1000research.141958.1 · Cited by 17
May the Source Be with You: On ChatGPT, Cybersecurity, and Secure Coding
Espinha, T., Iosif, A.-C., Kessba, I., Amburi, S., Lechner, U., & Pinto-Albuquerque, M. (2024) · Information · doi:10.3390/info15090572 · Cited by 15
Creating Human-Computer Partnerships
Mackay, W. E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-49425-3_1 · Cited by 10
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 19
Navigating (in)Security of AI-Generated Code
Ambati, S. H., Ridley, N., Branca, E., & Stakhanova, N. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679468 · Cited by 12
Security Evaluation of Companion Android Applications in IoT: The Case of Smart Security Devices
Allen, A., Mylonas, A., Vidalis, S., & Gritzalis, D. A. (2024) · Sensors · doi:10.3390/s24175465 · Cited by 10
Evaluating LLaMA 3.2 for Software Vulnerability Detection
Gonçalves, J., Silva, M., Cabral, B., Dias, T., Maia, E., Praça, I., Severino, R., & Ferreira, L. L. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94855-8_3 · Cited by 14
AI-Assisted Security: A Step towards Reimagining Software Development for a Safer Future
Shi, Y., Sakib, N., Shahriar, H., et al. (2023) · Annual International Computer Software and Applications Conference · doi:10.1109/compsac57700.2023.00142 · Cited by 4
Correctness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 39
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 10
LLM Critics Help Catch LLM Bugs
McAleese, N., Pokorny, R. M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., & Leike, J. (2024) · arXiv · doi:10.48550/arxiv.2407.00215 · Cited by 10
“Here CriticGPT’s comment points out a security error made by ChatGPT-4 when presented with a question from Perry et al. [22].”
BackgroundMethodAnalysis ChatGPT Potential: Transforming Software Development with AI Chat Bots
Purwoko, J. W., Abdullah, T., Wijaya, B., Gunawan, A. A. S., & Saputra, K. E. (2023) · International Conference on Networking, Electrical Engineering, Computer Science, and Technology (IConNECT) · doi:10.1109/iconnect56593.2023.10327087 · Cited by 7
The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who
Schlichtkrull, M., Ousidhoum, N. D., & Vlachos, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.577 · Cited by 15
Kudriavtseva, A., Hotak, N. A., & Gadyatskaya, O. (2025) · Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing · doi:10.1145/3672608.3707778 · Cited by 9
Exploring Student-AI Interactions in Vibe Coding
Geng, F., Shah, A., Li, H., Mulla, N., Swanson, S., Raj, G. S., Zingaro, D., & Porter, L. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786236 · Cited by 41
Dora, S., Lunkad, D., Aslam, N., Venkatesan, S., & Shukla, S. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-13714-2_3 · Cited by 8
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 14
Do AI Assistants Help Students Write Formal Specifications? A Study with ChatGPT and the B-Method
Capozucca, A., Yampolskyi, D., Goldberg, A., & Cristiá, M. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00009 · Cited by 8
Towards Secure Code Generation With LLMs: A Study on Common Weakness Enumeration
Zhao, J., Sun, Y., Huang, C., Liu, C., Guan, Y., Zeng, Y., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619281 · Cited by 17
Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
Schreiber, M., & Tippe, P. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-981-95-3537-8_9 · Cited by 6
Serafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 6
Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications
Sabouri, S., Eibl, P., Zhou, X., Ziyadi, M., Medvidović, N., Lindemann, L., & Chattopadhyay, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00199 · Cited by 17
Honarvar, S., van der Wilk, M., & Donaldson, A. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989005 · Cited by 8
Das, J. K., Mondal, S., & Roy, C. K. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00015 · Cited by 5
What Do Developers Want From AI?
D’Angelo, S., Murillo, A., Chandra, S., et al. (2024) · IEEE Software · doi:10.1109/ms.2024.3363538 · Cited by 7
Mapping the Trust Terrain: LLMs in Software Engineering—Insights and Perspectives
Khati, D., Liu, Y., Palacio, D. N., Zhang, Y., & Poshyvanyk, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771282 · Cited by 17
The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
Siddiq, M. L., Dristi, S., Saha, J., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00028 · Cited by 22
Metadata
- Authors
- Perry, N., Srivastava, M., Kumar, D., & Boneh, D.
- Affiliations
- Stanford University (US)
- University of California San Diego (US)
- Venue
- Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2023-11-15
- Pages
- 2785-2799
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W4388858772
- Cited by
- 259
- References
- 23
- Citations by year
- 2026: 99 · 2025: 83 · 2024: 55 · 2023: 22
- FWCI
- 66.42
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1656518)
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- software security, security vulnerabilities, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
