Transformer-Based Language Models for Software Vulnerability Detection
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
The large transformer-based language models demonstrate excellent performance in natural language processing. By considering the transferability of the knowledge gained by these models in one domain to other related domains, and the closeness of natural languages to high-level programming languages, such as C/C++, this work studies how to leverage (large) transformer-based language models in detecting software vulnerabilities and how good are these models for vulnerability detection tasks. In this regard, firstly, we present a systematic (cohesive) framework that details source code translation, model preparation, and inference. Then, we perform an empirical analysis of software vulnerability datasets of C/C++ source codes having multiple vulnerabilities corresponding to the library function call, pointer usage, array usage, and arithmetic expression. Our empirical results demonstrate the good performance of the language models in vulnerability detection. Moreover, these language models have better performance metrics, such as F1-score, than the contemporary models, namely bidirectional long short term memory and bidirectional gated recurrent unit. Experimenting with the language models is always challenging due to the requirement of computing resources, platforms, libraries, and dependencies. Thus, this paper also analyses the popular platforms to efficiently fine-tune these models and present recommendations while choosing the platforms for our framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“CodeBERT has a few variants, such as (i) GraphCodeBERT [12] that uses semantic-level data flow structure, (ii) BART [18], with both encoder and decoder part in its model architecture, and its extension (iii) PLBART [1].”
BackgroundLearning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 1×
“It trains on bimodal data (code & documents) [16], with codes from Python, Java, JavaScript, Ruby, Go, and PHP.”
Method
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Ferrag, M. A., Ndhlovu, M., Tihanyi, N., Cordeiro, L. C., Debbah, M., Lestable, T., & Thandi, N. S. (2024) · IEEE Access · doi:10.1109/access.2024.3363469 · Cited by 312
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
“1 Large Language Model for Code Generation Large Language Models have been emerging as the state-of-the-art in code-related tasks, advancing the progress on program understanding [12, 46], analysis [19, 22, 40], and generation [6, 18, 23].”
BackgroundGenerative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., & Stringhini, G. (2023) · arXiv · doi:10.48550/arxiv.2312.12575 · Cited by 132
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69
Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities
Khare, A., Dutta, S., Li, Z., Solko-Breslin, A., Alur, R., & Naik, M. (2023) · arXiv · doi:10.48550/arxiv.2311.16169 · Cited by 64
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
“…137:2 J. Shi et al. various programming languages, with exceptional performance across a spectrum of code-related tasks, ranging from code generation [13, 15, 143] and summarization [2, 3, 71] to vulnerability detection [97, 120, 159], program repair [57, 59, 142], and more [29, 90, 102, 106, 153].”
Smart Contract Vulnerability Detection: The Role of Large Language Model (LLM)
Boi, B., Esposito, C., & Lee, S. (2024) · ACM SIGAPP Applied Computing Review · doi:10.1145/3687251.3687253 · Cited by 41
“Addressing these challenges will be crucial in realizing the full potential of language model-based vulnerability analysis in practice [35].”
Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
Rigaki, M., Lukáš, O., Catania, C., & Garcia, S. (2024) · Proceedings of the 16th International Conference on Agents and Artificial Intelligence · doi:10.5220/0012391800003636 · Cited by 17
“This offers the potential to analyze software code and system configurations for vulnerabilities and prioritize fixes based on severity [25, 33].”
Background
Metadata
- Authors
- Thapa, C., Jang, S. I., Ahmed, M. E., Camtepe, S., Pieprzyk, J., & Nepal, S.
- Venue
- Proceedings of the 38th Annual Computer Security Applications Conference
- Publisher
- ACM
- Published
- 2022-12-05
- Pages
- 481-496
- ISSN
- 1063-9527 (ISSN-L)
- OpenAlex
- W4311165836
- Cited by
- 122
- References
- 9
- FWCI
- 22.11
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.