The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
SecureBERT: A Domain-Specific Language Model for Cybersecurity
Open accessAbstract
Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the world's economic and cultural infrastructures, while they have introduced risks in terms of cyber attacks. CTI is information that helps cybersecurity analysts make intelligent security decisions, that is often delivered in the form of natural language text, which must be transformed to machine readable format through an automated procedure before it can be used for automated security measures. This paper proposes SecureBERT, a cybersecurity language model capable of capturing text connotations in cybersecurity text (e.g., CTI) and therefore successful in automation for many critical cybersecurity tasks that would otherwise rely on human expertise and time-consuming manual efforts. SecureBERT has been trained using a large corpus of cybersecurity text.To make SecureBERT effective not just in retaining general English understanding, but also when applied to text with cybersecurity implications, we developed a customized tokenizer as well as a method to alter pre-trained weights. The SecureBERT is evaluated using the standard Masked Language Model (MLM) test as well as two additional standard NLP tasks. Our evaluation studies show that SecureBERT\footnote{\url{https://github.com/ehsanaghaei/SecureBERT}} outperforms existing similar models, confirming its capability for solving crucial NLP tasks in cybersecurity.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SecureBERT [1,2] is a specialized language model designed for cybersecurity applications, particularly effective in understanding and classifying cybersecurity-related texts, including those related to insider threats.”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
In the citing paper
“It was developed by fine-tuning the RoBERTa base model with a substantial dataset comprising 98,411 cybersecurity-related textual elements (equivalent to about 1 billion tokens).”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
In the citing paper
“It processes input sequences up to 512 tokens using 12 transformer layers with selfattention to capture deep contextual meaning.”
From User Behavior Analytics for Insider Threat using Transformer Based Approach (2026) · cites this paper as [2]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“2% accuracy on Stanford Question Answering Dataset (SQuAD) [23].”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 2×
“Beltagy et al. [7] unveiled SciBERT following the exact BERT’s architecture, a model that improves performance on downstream scientific NLP tasks by exploiting unsupervised pretraining from scratch on a 1 .”
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 1×
“ClinicalBERT [5] is another domain adaptation model based on BERT which is trained on clinical text from the MIMIC-III database.”
Training with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
Towards Robust Neural Networks via Random Self-ensemble
Liu, X., Cheng, M., Zhang, H., & Hsieh, C.-J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_23 · Cited by 386
Research works printed but not linked: 6
- Aghaei, E., Serpen, G.: Host-based anomaly detection using eigentraces feature extraction and one-class classification on system call trace data. J. Inf. Assurance Sec. (JIAS) 14(4), 106–117 (2019)
- Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) · arXiv
- Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference On Artificial Intelligence And Statistics, pp. 249–256. JMLR Workshop and Conference Proceedings (2010)
- Liu, Y., et al.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018) · arXiv
- Radford, A., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Cited by
Ferrag, M. A., Ndhlovu, M., Tihanyi, N., Cordeiro, L. C., Debbah, M., Lestable, T., & Thandi, N. S. (2024) · IEEE Access · doi:10.1109/access.2024.3363469 · Cited by 312
Zhou, H., Hu, C., Yuan, Y., Cui, Y., Jin, Y., Chen, C., Wu, H., Yuan, D., Jiang, L., Wu, D., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3465447 · Cited by 244
Large Language Models in Cybersecurity: State-of-the-Art
Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024) · arXiv · doi:10.48550/arxiv.2402.00891 · Cited by 92
“Within a particular programming context that the authors had defined, they tested scenarios with and without AI support.”
From Vulnerability to Defense: The Role of Large Language Models in Enhancing Cybersecurity
Kasri, W., Himeur, Y., Alkhazaleh, H. A., Tarapiah, S., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2025) · Computation · doi:10.3390/computation13020030 · Cited by 112
Large Language Models Meet Next-Generation Networking Technologies: A Review
Hang, C. N., Yu, P.-D., Morabito, R., & Tan, C. W. (2024) · Future Internet · doi:10.3390/fi16100365 · Cited by 93
Metadata
- Authors
- Aghaei, E., Niu, X., Shadid, W., & Al-Shaer, E.
- Affiliations
- University of North Carolina at Charlotte (US)
- Carnegie Mellon University (US)
- Corresponding author
- Aghaei, Ehsan
- Venue
- Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
- Publisher
- Springer Nature Switzerland
- Published
- 2023
- Pages
- 39-56
- ISSN
- 1867-8211 (ISSN-L)
- 1867-822X
- OpenAlex
- W4319079731
- Cited by
- 210
- References
- 26
- Citations by year
- 2026: 61 · 2025: 81 · 2024: 56 · 2023: 12
- FWCI
- 37.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- domain-specific language models, cybersecurity, cyber threat intelligence, masked language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SecureBERT: A Domain-Specific Language Model for Cybersecurity. https://app.cochat.ai/papers/10.1007/978-3-031-25538-0_3
- The paper
- Aghaei, E., Niu, X., Shadid, W., & Al-Shaer, E. (2023). SecureBERT: A Domain-Specific Language Model for Cybersecurity. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. https://doi.org/10.1007/978-3-031-25538-0_3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.