CoChat
Sign Up
PreprintCited by 5Software Engineering ResearchSoftware Testing and Debugging TechniquesSecurity and Verification in Computing

The DOI matches its DataCite record.

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Open access
Wang, Jiexin, Cao, Liuwen, Luo, Xitong, Cao, Yang, Li, Zhenghao, Xiao, Yunyi, Zhao, Mengchen, Jatowt, Adam, Cai, Yi·2024·arXiv
DOI 10.48550/arxiv.2407.02395arXiv 2407.02395
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.02395

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 32

  1. [1][1] Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019. JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, 5436-5446. doi:10.18653/v1/D19-1546 - · DOI
  2. [2][2] Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2023. Empirical analysis of security vulnerabilities in Python packages. Empirical Software Engineering 28, 3 (2023), 59. doi:10.1007/s10664-022-10278-4 - · DOI
  3. [3][3] Owura Asare, Meiyappan Nagappan, and N. Asokan. 2023. Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code? Empirical Software Engineering 28, 6 (2023), 129. doi:10.1007/s10664-023-10380-1 - · DOI · Paper page
  4. [4][4] Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. 2023. Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models. arXiv preprint arXiv:2312.04724 (2023). doi:10.48550/arXiv.2312.04724 - · DOI · Paper page
  5. [5][5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). doi:10.48550/arXiv.2107.03374 - · DOI · Paper page
  6. [7][7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024). doi:10.48550/arXiv.2407.21783 - · DOI · Paper page
  7. [8][8] Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2024. CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) . IEEE, 684-709. doi:10.1109/SaTML59370.2024.00040 - · DOI
  8. [9][9] Mohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, and Niranjan Balasubramanian. 2025. Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 26039-26057. doi:10.18653/v1/2025.acl-long.1263 - · DOI
  9. [10][10] Jingxuan He and Martin Vechev. 2023. Large Language Models for Code: Security Hardening and Adversarial Testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . Association for Computing Machinery, 1865-1879. doi:10.1145/3576915.3623175 - · DOI
  10. [11][11] Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction Tuning for Secure Code Generation. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235) . PMLR, 18043-18062. doi:10.5555/3692070.3692793 - · DOI
  11. [12][12] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) . doi:10.48550/arXiv.2105.09938 - · DOI · Paper page
  12. [13][13] Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. 2022. Fault-Aware Neural Code Rankers. Advances in Neural Information Processing Systems 35 (2022), 13419-13432. doi:10.52202/068431-0976 - · DOI
  13. [14][14] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping Language to Code in Programmatic Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 1643-1652. doi:10.18653/v1/D18-1192 - · DOI · Paper page
  14. [15][15] Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How Secure is Code Generated by ChatGPT?. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) . IEEE, 2445-2451. doi:10. 1109/SMC53992.2023.10394237 -
  15. [16][16] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202) . PMLR, 18319-18345. doi:10.5555/3618408.3619164 - · DOI
  16. [17][17] Bo Lin, Shangwen Wang, Yihao Qin, Liqian Chen, and Xiaoguang Mao. 2025. Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security . Association for Computing Machinery, 3356-3370. doi:10.1145/3719027. 3765049 - · DOI
  17. [18][18] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024). doi:10.48550/ arXiv.2412.19437 - · arXiv
  18. [19][19] Ana Nunez, Nafis Tanveer Islam, Sumit Kumar Jha, and Peyman Najafirad. 2024. AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing. arXiv preprint arXiv:2409.10737 (2024). doi:10.48550/arXiv.2409.10737 - · DOI · Paper page
  19. [20][20] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). doi:10.48550/arXiv.2303.08774 - · DOI · Paper page
  20. [21][21] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 311-318. doi:10.3115/1073083.1073135 - · DOI · Paper page
  21. [22][22] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 754-768. doi:10.1109/SP46214.2022.9833571 - · DOI · Paper page
  22. [23][23] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . Association for Computing Machinery, 2785-2799. doi:10.1145/3576915.3623157 - · DOI · Paper page
  23. [24][24] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv preprint arXiv:2009.10297 (2020). doi:10.48550/arXiv.2009.10297 - · DOI · Paper page
  24. [25][25] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). doi:10.48550/arXiv.2308.12950 - · DOI · Paper page
  25. [26][26] Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories. arXiv preprint arXiv:2504.21205 (2025). doi:10.48550/arXiv.2504.21205 - · DOI · Paper page
  26. [27][27] Jiahao Shi and Tianyi Zhang. 2025. RESCUE: Retrieval Augmented Secure Code Generation. arXiv preprint arXiv:2510.18204 (2025). doi:10.48550/arXiv.2510.18204 - · DOI
  27. [28][28] Nima Shiri Harzevili, Jiho Shin, Junjie Wang, Song Wang, and Nachiappan Nagappan. 2023. Automatic Static Bug Detection for Machine Learning Libraries: Are We There Yet? arXiv preprint arXiv:2307.04080 (2023). doi:10.48550/ arXiv.2307.04080 - · arXiv
  28. [29][29] Mohammed Latif Siddiq and Joanna C. S. Santos. 2022. SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security . Association for Computing Machinery, 29-33. doi:10.1145/3549035.3561184 - · DOI · Paper page
  29. [32][32] Catherine Tony, Markus Mutas, Nicolás E. Díaz Ferreyra, and Riccardo Scandariato. 2023. LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 588-592. doi:10.1109/MSR59073.2023.00084 - · DOI · Paper page
  30. [33][33] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824-24837. doi:10.52202/068431-1800 - · DOI · Paper page
  31. [34][34] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). doi:10.48550/arXiv.2505.09388 - · DOI · Paper page
  32. [35][35] Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. SecCoder: Towards Generalizable and Robust Secure Code Generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Miami, Florida, USA, 14557-14571. doi:10.18653/v1/2024.emnlp-main.806 Received 2026-01-30; accepted 2026-04-16 · DOI

Cited by

Sorted by citations

Metadata

Authors
Wang, J., Cao, L., Luo, X., Cao, Y., Li, Z., Xiao, Y., Zhao, M., Jatowt, A., & Cai, Y.
Affiliations
  • South China University of Technology (CN)
  • Universität Innsbruck (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
3
Issue
ISSTA
Pages
1558-1579
ISSN
2994-970X (ISSN-L)
DOI
10.48550/arxiv.2407.02395
arXiv
2407.02395
OpenAlex
W4400374531
Cited by
5
References
0
Citations by year
2026: 3 · 2025: 2
FWCI
25.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (62402185, 62476097)
  • Natural Science Foundation of Guangdong Province (2023B1515120078, 2024B1515040010)
  • Fundamental Research Funds for the Central Universities (62476097)
  • Science and Technology Planning Project of Guangdong Province (2024B1515040010, 2025B0101120003)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
Keywords
secure code generation, large language models, program synthesis, Python

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review