The DOI matches its DataCite record.
Finetuning Large Language Models for Vulnerability Detection
Open accessPreprint — published as 10.1109/access.2025.3546700
Abstract
This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Their results suggest that this is indeed the case and that LLMs have great potential for application in vulnerability detection.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Shestov et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 16
- Guru Bhandari, Amara Naseer, and Leon Moonen. Cvefixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering , PROMISE 2021, pp. 30-39, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386807. doi: 10.1145/3475960.3475985. URL https://doi.org/10.1145/3475960.3475985 . · DOI
- S. Chakraborty, R. Krishna, Y. Ding, and B. Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering , 48(09):3280-3296, sep 2022. ISSN 1939-3520. doi: 10.1109/TSE.2021.3087402. · DOI
- Roland Croft, M. Ali Babar, and M. Mehdi Kholoosi. Data quality for software vulnerability datasets. In Proceedings of the 45th International Conference on Software Engineering , ICSE '23, pp. 121-133. IEEE Press, 2023. ISBN 9781665457019. doi: 10.1109/ICSE48619.2023.00022. URL https://doi.org/10.1109/ICSE48619.2023.00022 . · DOI
- Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020 , pp. 1536-1547, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.139. URL https://aclanthology.org/2020.findings-emnlp.139 . · DOI · Paper page
- Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulnerability prediction. In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) , pp. 608-620, 2022. doi: 10.1145/3524842.3528452. · DOI
- Michael Fu, Chakkrit Kla Tantithamthavorn, Van Nguyen, and Trung Le. Chatgpt for vulnerability detection, classification, and repair: How far are we? In 2023 30th Asia-Pacific Software Engineering Conference (APSEC) , pp. 632-636, 2023. doi: 10.1109/APSEC60848.2023.00085. · DOI
- Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .
- Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ ar. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) , pp. 2999-3007, 2017. doi: 10.1109/ICCV.2017.324. · DOI · Paper page
- Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, and Yang Liu. Contrabert: Enhancing code pre-trained models via contrastive learning. In Proceedings of the 45th International Conference on Software Engineering , ICSE '23, pp. 2476-2487. IEEE Press, 2023a. ISBN 9781665457019. doi: 10.1109/ICSE48619.2023.00207. URL https://doi.org/10.1109/ICSE48619.2023.00207 . · DOI
- Zhihong Liu, Qing Liao, Wenchao Gu, and Cuiyun Gao. Software vulnerability detection with gpt and in-context learning. In 2023 8th International Conference on Data Science in Cyberspace (DSC) , pp. 229-236, 2023b. doi: 10.1109/DSC59305.2023.00041. · DOI
- Serena E. Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and C´ edric Dangremont. A manually-curated dataset of fixes to vulnerabilities of open-source software. In Proceedings of the 16th International Conference on Mining Software Repositories , MSR '19, pp. 383-387. IEEE Press, 2019. doi: 10.1109/MSR.2019.00064. URL https://doi.org/10.1109/MSR.2019.00064 . · DOI
- Moumita Das Purba, Arpita Ghosh, Benjamin J. Radford, and Bill Chu. Software vulnerability detection using large language models. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW) , pp. 112-119, 2023. doi: 10.1109/ISSREW60843. 2023.00058. · DOI
- Shichao Wang, Yun Zhang, Liagfeng Bao, Xin Xia, and Minghui Wu. Vcmatch: A ranking-based approach for automatic security patches localization for oss vulnerabilities. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , pp. 589-600, 2022. doi: 10.1109/SANER53432.2022.00076. · DOI
- J. Zhang, Z. Liu, X. Hu, X. Xia, and S. Li. Vulnerability detection by learning from syntax-based execution paths of code. IEEE Transactions on Software Engineering , 49(08):4196-4212, aug 2023. ISSN 1939-3520. doi: 10.1109/TSE.2023.3286586. · DOI
- Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACMSIGKDDConferenceon Knowledge Discovery and Data Mining , KDD '23, pp. 5673-5684, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305.3599790. URL https://doi.org/10.1145/3580305.3599790 . · DOI · Paper page
- Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead. ArXiv , abs/2404.02525, 2024. URL https://api.semanticscholar.org/CorpusID:268876353 . · arXiv
Cited by
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
Metadata
- Authors
- Shestov, A., Levichev, R., Mussabayev, R., Maslov, E., Cheshkov, A., & Zadorozhny, P.
- Affiliations
- Satbayev University (KZ)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 13
- Pages
- 38889-38900
- ISSN
- 2169-3536 (ISSN-L)
- arXiv
- 2401.17010
- OpenAlex
- W4408047557
- Cited by
- 93
- References
- 0
- Citations by year
- 2026: 36 · 2025: 48 · 2024: 9
- FWCI
- 87.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Ministry of Education and Science of the Republic of Kazakhstan (BR21882268)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- vulnerability detection, Java source code, large language models, StarCoder, code LLMs, fine-tuning, class imbalance, transfer learning, training efficiency, CodeBERT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Finetuning Large Language Models for Vulnerability Detection. https://app.cochat.ai/papers/10.48550/arxiv.2401.17010
- The paper
- Shestov, A., Levichev, R., Mussabayev, R., Maslov, E., Cheshkov, A., & Zadorozhny, P. (2024). Finetuning Large Language Models for Vulnerability Detection. arXiv. https://doi.org/10.48550/arxiv.2401.17010
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
