CoChat
Sign Up
PreprintCited by 8Software Engineering ResearchWeb Application Security VulnerabilitiesSoftware Testing and Debugging Techniques

LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward

Open access
Islam, Nafis Tanveer, Khoury, Joseph, Seong, Andrew, Karkevandi, Mohammad Bahrami, Parra, Gonzalo De La Torre, Bou-Harb, Elias, Najafirad, Peyman·2024·arXiv
DOI 10.48550/arxiv.2401.03374arXiv 2401.03374
Open this paper in CoChat — free accountOpen PDF

Abstract

In software development, the predominant emphasis on functionality often supersedes security concerns, a trend gaining momentum with AI-driven automation tools like GitHub Copilot. These tools significantly improve developers' efficiency in functional code development. Nevertheless, it remains a notable concern that such tools are also responsible for creating insecure code, predominantly because of pre-training on publicly available repositories with vulnerable code. Moreover, developers are called the "weakest link in the chain" since they have very minimal knowledge of code security. Although existing solutions provide a reasonable solution to vulnerable code, they must adequately describe and educate the developers on code security to ensure that the security issues are not repeated. Therefore we introduce a multipurpose code vulnerability analysis system \texttt{SecRepair}, powered by a large language model, CodeGen2 assisting the developer in identifying and generating fixed code along with a complete description of the vulnerability with a code comment. Our innovative methodology uses a reinforcement learning paradigm to generate code comments augmented by a semantic reward mechanism. Inspired by how humans fix code issues, we propose an instruction-based dataset suitable for vulnerability analysis with LLMs. We further identify zero-day and N-day vulnerabilities in 6 Open Source IoT Operating Systems on GitHub. Our findings underscore that incorporating reinforcement learning coupled with semantic reward augments our model's performance, thereby fortifying its capacity to address code vulnerabilities with improved efficacy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.03374

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SecRepair uses reinforcement learning with a semantic reward mechanism to improve the model's ability to generate accurate code comments and descriptions, guiding developers to address security issues.

    “It utilizes reinforcement learning with the semantic reward mechanism to improve the model's ability to generate accurate code comments and descriptions, guiding developers to address security issues.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Islam et al. (2024)

    The “Reinforcement Learning for Code Comment” section says SecRepair uses a semantic reward to fine-tune generation of code comments from developer-friendly descriptions. It says those descriptions are generated before that step, so it does not establish that reinforcement learning improves description generation too. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. Ahmad, A.; Waseem, M.; Liang, P.; Fahmideh, M.; Aktar, M. S.; and Mikkonen, T. 2023. Towards human-bot collab- orative software architecting with chatgpt. In Proceedings of the 27th International Conference on Evaluation and As- sessment in Software Engineering, 279-285.
  2. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
  3. Chen, Z.; Kommrusch, S.; and Monperrus, M. 2022. Neural transfer learning for repairing security vulnerabilities in c code. IEEE Transactions on Software Engineering, 49(1): 147-165.
  4. Eichholz, M.; Campbell, E.; Foster, N.; Salvaneschi, G.; and Mezini, M. 2019. How to avoid making a billion-dollar mis- take: Type-safe data plane programming with SafeP4. arXiv preprint arXiv:1906.07223. · arXiv
  5. Freitag, M.; and Al-Onaizan, Y. 2017. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806. · arXiv
  6. Fu, M.; Tantithamthavorn, C.; Le, T.; Nguyen, V.; and Phung, D. 2022. VulRepair: a T5-based automated soft- ware vulnerability repair. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, 935- 947.
  7. Hallak, A.; Di Castro, D.; and Mannor, S. 2015. Con- textual markov decision processes. arXiv preprint arXiv:1502.02259. · arXiv
  8. He, J.; and Vechev, M. 2023. Controlling large language models to generate secure and vulnerable code. arXiv preprint arXiv:2302.05319. · arXiv
  9. Huang, Z.; Lie, D.; Tan, G.; and Jaeger, T. 2019. Using safety properties to generate vulnerability patches. In 2019 IEEE Symposium on Security and Privacy (SP), 539-554.
  10. Jang, J.; Agrawal, A.; and Brumley, D. 2012. ReDeBug: finding unpatched code clones in entire os distributions. In 2012 IEEE Symposium on Security and Privacy, 48-62.
  11. Jesse, K.; Ahmed, T.; Devanbu, P. T.; and Morgan, E. 2023. Large Language Models and Simple, Stupid Bugs. arXiv preprint arXiv:2303.11455. · arXiv
  12. Kim, S.; Woo, S.; Lee, H.; and Oh, H. 2017. Vuddy: A scalable approach for vulnerable code clone discovery. In 2017 IEEE Symposium on Security and Privacy (SP), 595- 614. IEEE.
  13. Li, Z.; Zou, D.; Xu, S.; Chen, Z.; Zhu, Y.; and Jin, H. 2021. Vuldeelocator: a deep learning-based fine-grained vulnera- bility detector. IEEE Transactions on Dependable and Se- cure Computing.
  14. Murali, V.; Maddila, C.; Ahmad, I.; Bolin, M.; Cheng, D.; Ghorbani, N.; Fernandez, R.; and Nagappan, N. 2023. CodeCompose: A Large-Scale Industrial Deployment of AI- assisted Code Authoring. arXiv preprint arXiv:2305.12050. · arXiv
  15. Nijkamp, E.; Hayashi, H.; Xiong, C.; Savarese, S.; and Zhou, Y. 2023. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309. · arXiv
  16. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine trans- lation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311-318.
  17. Pearce, H.; Ahmad, B.; Tan, B.; Dolan-Gavitt, B.; and Karri, R. 2022a. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Sympo- sium on Security and Privacy (SP), 754-768. IEEE.
  18. Pearce, H.; Tan, B.; Ahmad, B.; Karri, R.; and Dolan-Gavitt, B. 2022b. Examining Zero-Shot Vulnerability Repair with Large Language Models. In 2023 IEEE Symposium on Se- curity and Privacy (SP), 1-18. IEEE Computer Society.
  19. Prenner, J. A.; and Robbes, R. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922. · arXiv
  20. Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. Zero: Memory optimizations toward training trillion param- eter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1-16. IEEE.
  21. Ross, S. I.; Martinez, F.; Houde, S.; Muller, M.; and Weisz, J. D. 2023. The programmer's assistant: Conversational in- teraction with a large language model for software develop- ment. In Proceedings of the 28th International Conference on Intelligent User Interfaces, 491-514.
  22. Sandoval, G.; Pearce, H.; Nys, T.; Karri, R.; Dolan-Gavitt, B.; and Garg, S. 2022. Security Implications of Large Lan- guage Model Code Assistants: A User Study. arXiv preprint arXiv:2208.09727. · arXiv
  23. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. · arXiv
  24. Wang, Y.; Wang, W.; Joty, S.; and Hoi, S. C. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Mod- els for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8696-8708. Online and Punta Cana, Dominican Republic: Association for Computational Lin- guistics. WhiteHouse. 2023. Executive Order on Improving the Na- tion's Cybersecurity. Accessed: 2023-07-27.
  25. Woo, S.; Hong, H.; Choi, E.; and Lee, H. 2022. {MOVERY}: A Precise Approach for Modified Vulnerable Code Clone Discovery from Modified {Open-Source} Soft- ware Components. In 31st USENIX Security Symposium (USENIX Security 22), 3037-3053.
  26. Xiao, Y.; Chen, B.; Yu, C.; Xu, Z.; Yuan, Z.; Li, F.; Liu, B.; Liu, Y.; Huo, W.; Zou, W.; et al. 2020. {MVP}: Detecting Vulnerabilities using {Patch-Enhanced} Vulnerability Sig- natures. In 29th USENIX Security Symposium (USENIX Se- curity 20), 1165-1182.
  27. Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. · arXiv
  28. Zhang, Y.; Gao, X.; Duck, G. J.; and Roychoudhury, A. 2022. Program vulnerability repair via inductive inference. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 691-702.

Cited by

Sorted by citations

Authors
Islam, N. T., Khoury, J., Seong, A., Karkevandi, M. B., Parra, G. D. L. T., Bou-Harb, E., & Najafirad, P.
Affiliations
  • The University of Texas at San Antonio (US)
  • Louisiana State University (US)
ORCID
Khoury, Joseph
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.03374
arXiv
2401.03374
OpenAlex
W4390723636
Cited by
8
References
0
Citations by year
2025: 5 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
large language models, reinforcement learning, code comments, vulnerability descriptions, zero-day vulnerabilities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review