Large Language Models for Code: Security Hardening and Adversarial Testing
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Large language models (large LMs) are increasingly trained on massive codebases and used to generate code. However, LMs lack awareness of security and are found to frequently produce unsafe code. This work studies the security of LMs along two important axes: (i) security hardening, which aims to enhance LMs' reliability in generating secure code, and (ii) adversarial testing, which seeks to evaluate LMs' security at an adversarial standpoint. We address both of these by formulating a new security task called controlled code generation. The task is parametric and takes as input a binary property to guide the LM to generate secure or unsafe code, while preserving the LM's capability of generating functionally correct code. We propose a novel learning-based approach called SVEN to solve this task. SVEN leverages property-specific continuous vectors to guide program generation towards the given property, without modifying the LM's weights. Our training procedure optimizes these continuous vectors by enforcing specialized loss terms on different regions of code, using a high-quality dataset carefully curated by us. Our extensive evaluation shows that SVEN is highly effective in achieving strong security control. For instance, a state-of-the-art CodeGen LM with 2.7B parameters generates secure code for 59.1% of the time. When we employ SVEN to perform security hardening (or adversarial testing) on this LM, the ratio is significantly boosted to 92.3% (or degraded to 36.8%). Importantly, SVEN closely matches the original LMs in functional correctness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“Although LMs excel in functional correctness, they may produce code with security issues [25, 27, 74].”
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
(2021) · doi:10.1109/tse.2019.2946563 · Cited by 574
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · cited 2×
“Moreover, to the best of our knowledge, SVEN is the first work to provide a realistic adversarial evaluation for LMs of code, under the constraint of preserving functional correctness for imperceptibility.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · cited 1×
“Continuous prompting is widely used for cost-effectively adapting LMs to different NLP tasks [37, 48, 49, 54, 62].”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243
GeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 7×
“For instance, the user can use SVENsec to harden open-source LMs [35, 57, 69].”
BackgroundMethodInfluential(2023) · doi:10.1109/sp46215.2023.10179324 · Cited by 131
Siddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101 · cited 10×
“Evaluating Security To assess the security of our models, we adopt the state-of-the-art methodology in [60, 68], which involves a diverse set of manually constructed scenarios that reflect real-world coding.”
BackgroundMethodInfluentialControllable Natural Language Generation with Contrastive Prefixes
Qian, J., Dong, L., Shen, Y., Wei, F., & Chen, W. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.229 · Cited by 74
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52 · cited 5×
“Recent user studies from academia confirmed the benefits of Copilot on increasing coding productivity, such as offering a useful starting point [68] and assisting users to write functionally correct code [62].”
InfluentialEfficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47 · cited 1×
“Both InCoder and SantaCoder were trained with the fill-in-the-middle objective [21], while CodeGen only involved standard left-to-right training.”
MethodMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Research works printed but not linked: 29
- 10.1109/SP46214.2022.9833659 · DOI
- Guru Prasad Bhandari Amara Naseer and Leon Moonen. 2021. CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-source Software. In PROMISE. https://doi.org/10.1145/3475960.3475985 · DOI · Paper page
- 10.1109/TSE.2021.3087402 · DOI · Paper page
- 10.1109/TSE.2022.3147265 · DOI · Paper page
- 10.1109/ICSE48619.2023.00022 · DOI · Paper page
- 10.18653/v1/n19--1423 · doi:10.18653/v1/n19--1423
- 10.1109/SP.2016.15 · DOI · Paper page
- 10.1145/3379597.3387501 · DOI · Paper page
- Luca Gazzola Daniela Micucci and Leonardo Mariani. 2018. Automatic Software Repair: a Survey. In ICSE. https://doi.org/10.1145/3180155.3182526 · DOI · Paper page
- 10.1109/MS.2021.3072577 · DOI
- Jingxuan He Luca Beurer-Kellner and Martin Vechev. 2022. On Distribution Shift in Learning-based Bug Detectors. In ICML. https://proceedings.mlr.press/v162/he22a.html
- 10.1162/coli_a_00426 · DOI · Paper page
- Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How Secure is Code Generated by ChatGPT? CoRR, Vol. abs/2304.09655 (2023). https://arxiv.org/abs/2304.09655 · arXiv
- Tomasz Korbak Hady Elsahar Germán Kruszewski and Marc Dymetman. 2022. Controlling Conditional Language Models without Catastrophic Forgetting. In ICML Kamalika Chaudhuri Stefanie Jegelka Le Song Csaba Szepesvári Gang Niu and Sivan Sabato (Eds.). https://proceedings.mlr.press/v162/korbak22a.html
- Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html
- 10.1109/TDSC.2021.3051525 · DOI · Paper page
- Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In NDSS. http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2018/02/ndss2018_03A-2_Li_paper.pdf
- 10.1109/JPROC.2020.2993293 · DOI · Paper page
- Georgios Nikitopoulos Konstantina Dritsa Panos Louridas and Dimitris Mitropoulos. 2021. CrossVul: a Cross-language Vulnerability Dataset with Commit Data. In ESEC/FSE. https://doi.org/10.1145/3468264.3473122 · DOI
- Yu Nong Yuzhe Ou Michael Pradel Feng Chen and Haipeng Cai. 2022. Generating Realistic Vulnerabilities via Neural Code Editing: an Empirical Study. In ESEC/FSE. https://doi.org/10.1145/3540250.3549128 · DOI
- Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining Zero-Shot Vulnerability Repair with Large Language Models. In IEEE S&P. https://doi.ieeecomputersociety.org/10.1109/SP46215.2023.00001 · doi:10.1109/sp46215.2023.00001
- Sofia Reis and Rui Abreu. 2021. A Ground-truth Dataset of Real Security Patches. CoRR, Vol. abs/2110.09635 (2021). https://arxiv.org/abs/2110.09635 · arXiv
- Roei Schuster Congzheng Song Eran Tromer and Vitaly Shmatikov. 2021. You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. In USENIX Security. https://www.usenix.org/conference/usenixsecurity21/presentation/schuster
- Justin Smith Brittany Johnson Emerson R. Murphy-Hill Bill Chu and Heather Richter Lipford. 2015. Questions Developers Ask While Diagnosing Potential Security Vulnerabilities with Static Analysis. In ESEC/FSE. https://doi.org/10.1145/2786805.2786812 · DOI
- Zhensu Sun Xiaoning Du Fu Song Mingze Ni and Li Li. 2022. CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning. In WWW. https://doi.org/10.1145/3485447.3512225 · DOI · Paper page
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
- 10.1016/j.infsof.2021.106809 · DOI · Paper page
- Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. 2022. FIXREVERTER: A Realistic Bug Injection Methodology for Benchmarking Fuzz Testing. In USENIX Security. https://www.usenix.org/conference/usenixsecurity22/presentation/zhang-zenong
- Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In NeurIPS. https://proceedings.neurips.cc/paper/2019/hash/49265d2447bc3bbfe9e76306ce40a31f-Abstract.html
Cited by
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
“He et al. [97] focused on enhancing the security of code generated by LLMs.”
BackgroundWhen LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“He and Vechev (2023) takes a more direct approach to customize LLMs through specific mechanisms.”
Sai, S., Yashvardhan, U., Chamola, V., & Sikdar, B. (2024) · IEEE Access · doi:10.1109/access.2024.3385107 · Cited by 125
Large Language Models in Cybersecurity: State-of-the-Art
Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024) · arXiv · doi:10.48550/arxiv.2402.00891 · Cited by 92
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2023) · arXiv · doi:10.48550/arxiv.2310.02059 · Cited by 75
Metadata
- Authors
- He, J., & Vechev, M.
- Venue
- Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2023-11-15
- Pages
- 1865-1879
- OpenAlex
- W4320560161
- Cited by
- 138
- References
- 34
- FWCI
- 26.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.