SecureBERT: A Domain-Specific Language Model for Cybersecurity
Open accessThe DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Abstract
Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the world's economic and cultural infrastructures, while they have introduced risks in terms of cyber attacks. CTI is information that helps cybersecurity analysts make intelligent security decisions, that is often delivered in the form of natural language text, which must be transformed to machine readable format through an automated procedure before it can be used for automated security measures. This paper proposes SecureBERT, a cybersecurity language model capable of capturing text connotations in cybersecurity text (e.g., CTI) and therefore successful in automation for many critical cybersecurity tasks that would otherwise rely on human expertise and time-consuming manual efforts. SecureBERT has been trained using a large corpus of cybersecurity text.To make SecureBERT effective not just in retaining general English understanding, but also when applied to text with cybersecurity implications, we developed a customized tokenizer as well as a method to alter pre-trained weights. The SecureBERT is evaluated using the standard Masked Language Model (MLM) test as well as two additional standard NLP tasks. Our evaluation studies show that SecureBERT\footnote{\url{https://github.com/ehsanaghaei/SecureBERT}} outperforms existing similar models, confirming its capability for solving crucial NLP tasks in cybersecurity.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SecureBERT [1,2] is a specialized language model designed for cybersecurity applications, particularly effective in understanding and classifying cybersecurity-related texts, including those related to insider threats.”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
In the citing paper
“It was developed by fine-tuning the RoBERTa base model with a substantial dataset comprising 98,411 cybersecurity-related textual elements (equivalent to about 1 billion tokens).”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
In the citing paper
“It processes input sequences up to 512 tokens using 12 transformer layers with selfattention to capture deep contextual meaning.”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Since BERTś goal is to generate a language model, only the encoder mechanism is necessary [27].”
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Inasimilar work on biomedical domain, Gu et al. [15] introduced BioBERT focusing particularly on biomedical domain using BERT architecture and publicly available biomedical datasets.”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“Socher et al. [26] used Amazon’s Mechanical Turk to create fine-grained labels for all parsed phrases in the corpus.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“2% accuracy on Stanford Question Answering Dataset (SQuAD) [23].”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“The pre-trained BERT model used Books Corpus (800M words) and English Wikipedia (2,500M words) and improved the state-of-the-art for eleven NLP tasks such as getting a GLUE [28] score of 80 .”
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 2×
“Beltagy et al. [7] unveiled SciBERT following the exact BERT’s architecture, a model that improves performance on downstream scientific NLP tasks by exploiting unsupervised pretraining from scratch on a 1 .”
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 1×
“ClinicalBERT [5] is another domain adaptation model based on BERT which is trained on clinical text from the MIMIC-III database.”
Training with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
Towards Robust Neural Networks via Random Self-ensemble
Liu, X., Cheng, M., Zhang, H., & Hsieh, C.-J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_23 · Cited by 386
Zur, R. M., Jiang, Y., Pesce, L. L., & Drukker, K. (2009) · Medical Physics · doi:10.1118/1.3213517 · Cited by 208 · cited 1×
“Referring to previous works on maintaining robust neural networks [18,31,33], incorporation of noise to an unstable neural network model with a limited training set can act as a regularizer and help reduce overfitting during the training.”
Neural Machine Translation with Byte-Level Subwords
Wang, C., Cho, K., & Gu, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6451 · Cited by 147
Yin, J., Tang, M., Cao, J., & Wang, H. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106529 · Cited by 132 · cited 1×
“Yin et al. [30] fine-tuned pre-trained BERT against cybersecurity text and developed a classifi-cation layer on top of their model, ExBERT, to extract sentence-level semantic features and predict the exploitability of vulnerabilities.”
Das, S. S., Serra, E., Halappanavar, M. M., Pothen, A., & Al-Shaer, E. S. (2021) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa53316.2021.9564227 · Cited by 104
Self-citationMalwareTextDB: A Database for Annotated Malware Articles
Lim, S. K., Muis, A. O., Lu, W., & Ong, C. H. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1143 · Cited by 94 · cited 1×
“MalwareTextDB [17] is a dataset containing 39 annotated APT reports with a total of 6,819 sentences.”
Adversarial Noise Layer: Regularize Neural Network by Adding Noise
You, Z., Ye, J., Li, K., Xu, Z., & Wang, P. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8803055 · Cited by 80 · cited 1×
“Referring to previous works on maintaining robust neural networks [18,31,33], incorporation of noise to an unstable neural network model with a limited training set can act as a regularizer and help reduce overfitting during the training.”
CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model
Ameri, K., Hempel, M., Sharif, H., Lopez Jr., J., & Perumalla, K. (2021) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp1040031 · Cited by 76 · cited 1×
“CyBERT [6] presents a classifier for cybersecurity feature claims by fine-tuning a pre-trained BERT language model for the purpose of identifying cybersecurity claims from a large pool of sequences in ICS device documents.”
An Intelligent Transient Stability Assessment Framework with Continual Learning Ability
Li, X., Yang, Z., Guo, P., & Cheng, J. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3064052 · Cited by 62 · cited 1×
“When a model is being trained from scratch, typically noise can be added to the hidden layers at each iteration, whereas in continual learning, it can be introduced to input data to generalize the model and reduce error [4,16].”
Data and knowledge-driven named entity recognition for cyber security
Gao, C., Zhang, X., & Liu, H. (2021) · Cybersecurity · doi:10.1186/s42400-021-00072-y · Cited by 57 · cited 1×
“Additionally, there are several studies on fine-tuning BERT for NER tasks such as [9], [32] and [13].”
ThreatZoom: Hierarchical Neural Network for CVEs to CWEs Classification
Aghaei, E., Shadid, W., & Al-Shaer, E. S. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-63086-7_2 · Cited by 46
Self-citationNamed Entity Recognition Using BERT with Whole World Masking in Cybersecurity Domain
Zhou, S., Liu, J., Zhong, X., & Zhao, W. (2021) · 2021 IEEE 6th International Conference on Big Data Analytics (ICBDA) · doi:10.1109/icbda51983.2021.9403180 · Cited by 42 · cited 1×
“Additionally, there are several studies on fine-tuning BERT for NER tasks such as [9], [32] and [13].”
Joint BERT Model based Cybersecurity Named Entity Recognition
Chen, Y., Ding, J., Li, D., & Chen, Z. (2021) · International Conference on Software Engineering and Information Management (ICSIM) · doi:10.1145/3451471.3451508 · Cited by 30 · cited 1×
“Additionally, there are several studies on fine-tuning BERT for NER tasks such as [9], [32] and [13].”
Aghaei, E., & Al-Shaer, E. S. (2019) · doi:10.1145/3314058.3318167 · Cited by 24 · cited 1×
“When it comes to cybersecurity tasks including cyber threat intelligence, vulnerability analysis, and threat action extraction [1,3], such knowledge is crucial and utilizing a model with SecureBERT’s properties would be highly beneficial.”
Self-citationDodgeTron: Towards Autonomous Cyber Deception Using Dynamic Hybrid Analysis of Malware
Sajid, M. S. I., Wei, J., Alam, M. R., Aghaei, E., & Al-Shaer, E. S. (2020) · IEEE Conference on Communications and Network Security · doi:10.1109/cns48642.2020.9162202 · Cited by 15 · cited 1×
“SecureBERT is generic enough to be applied in a variety of cybersecurity tasks, such as phishing detection [10], code and malware analysis [24], intrusion detection [2], etc. SecureBERT is a pre-trained cybersecurity language model that have the fundamental understanding of both the word-level and…”
Self-citationActive Defense Against Social Engineering: The Case for Human Language Technology
Dalton, A., Aghaei, E., Al-Shaer, E. S., Bhatia, A., Castillo, E., Cheng, Z., Dhaduvai, S., Duan, Q., Hebenstreit, B., Islam, M. M., et al. (2020) · Symposium on the Theory of Computing · Cited by 5
Self-citation
Research works printed but not linked: 6
- Aghaei, E., Serpen, G.: Host-based anomaly detection using eigentraces feature extraction and one-class classification on system call trace data. J. Inf. Assurance Sec. (JIAS) 14(4), 106–117 (2019)
- Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) · arXiv
- Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference On Artificial Intelligence And Statistics, pp. 249–256. JMLR Workshop and Conference Proceedings (2010)
- Liu, Y., et al.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018) · arXiv
- Radford, A., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Cited by
Ferrag, M. A., Ndhlovu, M., Tihanyi, N., Cordeiro, L. C., Debbah, M., Lestable, T., & Thandi, N. S. (2024) · IEEE Access · doi:10.1109/access.2024.3363469 · Cited by 312
“TABLE I provides a comparison of various recent works on cyber threat detection in terms of four key parameters: The majority of the research conducted in 2022, including [6], [11], and [10], integrated LLMs, but they did not support detection, nor did they utilize packet data.”
MethodZhou, H., Hu, C., Yuan, Y., Cui, Y., Jin, Y., Chen, C., Wu, H., Yuan, D., Jiang, L., Wu, D., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3465447 · Cited by 244
“Comparisons between SecureBERT and RoBERTa in masked tasks [121] Task 1: “Information from these scans may reveal opportunities for other forms < mask > establishing operational resources, or initial access.””
BackgroundMethodInfluentialLarge Language Models in Cybersecurity: State-of-the-Art
Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024) · arXiv · doi:10.48550/arxiv.2402.00891 · Cited by 92
“Within a particular programming context that the authors had defined, they tested scenarios with and without AI support.”
From Vulnerability to Defense: The Role of Large Language Models in Enhancing Cybersecurity
Kasri, W., Himeur, Y., Alkhazaleh, H. A., Tarapiah, S., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2025) · Computation · doi:10.3390/computation13020030 · Cited by 70
Large Language Models Meet Next-Generation Networking Technologies: A Review
Hang, C. N., Yu, P.-D., Morabito, R., & Tan, C. W. (2024) · Future Internet · doi:10.3390/fi16100365 · Cited by 93
“SecureBERT in [162] is a cybersecurity language model trained on a large corpus of cybersecurity text to automate critical tasks in Cyber Threat Intelligence by capturing text connotations and transforming natural language text into machine-readable formats.”
Metadata
- Authors
- Aghaei, E., Niu, X., Shadid, W., & Al-Shaer, E.
- Affiliations
- University of North Carolina at Charlotte (US)
- Carnegie Mellon University (US)
- Corresponding author
- Aghaei, Ehsan
- Venue
- Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
- Publisher
- Springer Nature Switzerland
- Published
- 2023
- Pages
- 39-56
- ISSN
- 1867-8211 (ISSN-L)
- 1867-822X
- OpenAlex
- W4319079731
- Cited by
- 210
- References
- 26
- Citations by year
- 2026: 61 · 2025: 81 · 2024: 56 · 2023: 12
- FWCI
- 37.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- domain-specific language models, cybersecurity, cyber threat intelligence, masked language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.