The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index; 17 of the 49 sources its claims cite were checked for support.
My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security
Open accessAbstract
Background: Generative AI (GAI) tools like GitHub Copilot and ChatGPT are transforming software development by automating code generation and enhancing developers' productivity. However, since these tools are often trained on open-source repositories, they may inadvertently reproduce vulnerable code, raising concerns about the security of AI-generated outputs. Aims: In this paper, we aim to investigate how developers perceive code security when using GAI tools. Method: We conducted a survey with 105 software developers with diverse experience levels to gather their perceptions regarding the security of generated code and their suggestions for improving it. Results: While developers reported increased development speed when using GAI tools, many spend additional time on security reviews and documentation of the generated code, and they are worried about the overreliance on AI and vulnerabilities in the code. Only about a quarter of the developers expressed confidence in the code generated by AI, and, moreover, experienced developers perceive that their proficiency in secure coding decreases when using GAI tools. Our results provide organizations with a better understanding of the risks associated with GAI tools and help improve their software security programs.
Overview
CoChat Summary
## TL;DR
This survey study of 105 developers examines how people perceive the security impact of generative AI code tools like ChatGPT and GitHub Copilot. Developers reported productivity gains, but many also reported more security review, low confidence in AI-generated code, and concerns about overreliance and vulnerabilities.
## Key findings
- Among 105 valid respondents, 78% named faster development as a benefit and 65% cited increased efficiency; 36% also saw improved vulnerability detection and 32% improved maintainability.
- 61% said they spend less time writing code with GAI tools, but 44% said they spend more time on security reviews, and 49% said documenting generated code is more important.
- Perceived risks were led by overreliance on AI without human oversight (65%), code with similar vulnerabilities (59%), introduction of new vulnerabilities (52%), and privacy/ethical concerns (48%).
- Only 25% were confident in the security of AI-generated code, while 43% lacked confidence; 89% reported using ChatGPT and 59% GitHub Copilot, and 62% of IntelliCode users were confident in its security.
## Methods
- The authors ran an anonymous survey using Qualtrics from May 19 to July 13, 2024, targeting software developers; the study was approved by Leiden University’s Science Ethics Committee.
- They received 196 responses and kept 105 valid responses; participants were grouped by experience into junior (student/learning, 0–2 years), medior (3–5 years), and senior (more than 6 years).
- Analysis combined descriptive statistics, open-ended theme extraction, and statistical tests: two Kruskal-Wallis tests and one Brunner-Munzel test, with Bonferroni correction at p = 0.016.
## Limitations
- The paper states the sample may have self-selection bias and non-response bias, and that the findings may not generalize to the whole developer population because the sample size was 105.
- The survey format limited follow-up questions and multiselect options, which may have constrained depth; this is stated in the paper.
- The study relied on self-reported answers, so misreporting or misunderstanding is possible; this is stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
14 Valid · 2 Not valid · 15 Not checked · 100 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, since GAIs are often trained on data from open-source repositories like GitHub, they may inadvertently learn and replicate code that contains software bugs and security vulnerabilities [46].”
What the sources say
- Supportedcites [46] [46] Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie, Adam Jatowt, and Yi Cai. 2024. Is Your AI-Generated Code Really Secure? Evaluating Large Language Models on Secure Code Generation with CodeSecEval. arXiv preprint arXiv:2407.02395 (2024). - source
The Introduction (p. 2) explicitly says LLMs are often trained on open-source repositories such as GitHub and may learn and reproduce code containing bugs and security vulnerabilities. This directly supports the attributed claim about AI code-generation tools. (LLM, conf=0.95)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Thus, our work independently confirms [21].”
What the sources say
- Supportedcites [21] [21] Jan H Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, Heather Richter Lipford, et al. 2024. Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns. - source
The citation identifies Klemmer et al. as the prior study; the source’s Key Findings: Security Concerns & Considerations section reports that study’s findings. Whether the citing authors’ work independently confirms those findings is their own conclusion, not a claim attributed to Klemmer et al. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“experienced developers perceive that their proficiency in secure coding decreases when using GAI tools.”
- Also stated in: § Abstract · plain
“experienced developers perceive that their proficiency in secure coding decreases when using GAI tools”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While developers reported increased development speed when using GAI tools,”
- Also stated in: § Abstract · plain
“While developers reported increased development speed when using GAI tools”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only about a quarter of the developers expressed confidence in the code generated by AI,”
- Also stated in: § Abstract · plain
“Only about a quarter of the developers expressed confidence in the code generated by AI”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § Abstract · plain
41 more claims
Create a free account to see all 131 claims and to verify the 15 references still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 46
- The quote only partly states this claim. 4
Citations
Cites
Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study
Deutskens, E., de Ruyter, K., Wetzels, M., & Oosterveld, P. (2004) · Marketing Letters · doi:10.1023/b:mark.0000021968.86465.00 · Cited by 835
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
The limitations of online surveys
Andrade, C. (2020) · Indian Journal of Psychological Medicine · doi:10.1177/0253717620957496 · Cited by 618
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Lies, Damned Lies, and Survey Self-Reports? Identity as a Cause of Measurement Bias
Brenner, P. S., & DeLamater, J. (2016) · Social Psychology Quarterly · doi:10.1177/0190272516628298 · Cited by 457 · cited 1×
“That means that participants could give wrong or incomplete answers about themselves because they want to appear better than they are or if they misunderstand the questions [7].”
Balakrishnan, J., Abed, S. S., & Jones, P. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.121692 · Cited by 307
Conversational commerce: entering the next stage of AI-powered digital assistants
Balakrishnan, J., & Dwivedi, Y. K. (2024) · Annals of Operations Research · doi:10.1007/s10479-021-04049-5 · Cited by 301
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217 · cited 1×
“5 https://www.qualtrics.com/ To construct the multiple-choice questions, we relied on the existing literature on developers’ behaviors, potential benefits, challenges, and risks associated with using AI-generated code, as well as factors contributing to security [8, 34, 35, 42, 45].”
Research works printed but not linked: 1
- Johan Linåker, Sardar Muhammad Sulaman, Rafael Maiani de Mello, and Martin Höst. 2015. Guidelines for conducting surveys in software engineering. (2015).
Cited by
On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
Baltes, S., Speith, T., Chiteri, B., Mohsenimofidi, S., Chakraborty, S., & Buschek, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659804 · Cited by 5
Paz-Chen, M., Lanir, J., & Ayalon, O. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2654495 · Cited by 1
Zare, A., & Assal, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29723-5_35
A comprehensive analysis of security risks and productivity impacts of AI-generated code
Zardari, S., Osama, M., Ammar, S. M., Reshamwala, R. I., & Siddiqui, A. (2026) · AI and Ethics · doi:10.1007/s43681-026-01203-2 · Cited by 1
Beyond syntax: enhancing automated documentation with data differences
Fantino, G., Vetrò, A., Torchiano, M., & Cappelluti, F. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00623-y · Cited by 1
Metadata
- Authors
- Kudriavtseva, A., Hotak, N. A., & Gadyatskaya, O.
- Affiliations
- Leiden University (NL)
- Venue
- Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing
- Publisher
- ACM
- Published
- 2025-03-31
- Pages
- 1637-1646
- OpenAlex
- W4410357414
- Cited by
- 9
- References
- 25
- Citations by year
- 2026: 9
- FWCI
- 10.74
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Symposium on Applied Computing (publishedVersion, cc-by-nc-sa)
- Leiden Repository (Leiden University) (submittedVersion, cc-by-sa)
- Funders
- NWO (NWA.1215.18.008)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- generative AI, code generation tools, GitHub Copilot, ChatGPT, software security, developer perceptions, secure coding, security review, developer experience
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security. https://app.cochat.ai/papers/10.1145/3672608.3707778
- The paper
- Kudriavtseva, A., Hotak, N. A., & Gadyatskaya, O. (2025). My Code Is Less Secure with Gen AI: Surveying Developers' Perceptions of the Impact of Code Generation Tools on Security. Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing. https://doi.org/10.1145/3672608.3707778
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.