Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 405 of the 405 references it cites were found in a registry or an index; 63 of the 193 sources its claims cite were checked for support.

When LLMs meet cybersecurity: a systematic literature review

Open access

Abstract

The rapid development of large language models (LLMs) has opened new avenues across various fields, including cybersecurity, which faces an evolving threat landscape and demand for innovative technologies. Despite initial explorations into the application of LLMs in cybersecurity, there is a lack of a comprehensive overview of this research area. This paper addresses this gap by providing a systematic literature review, covering the analysis of over 300 works, encompassing 25 LLMs and more than 10 downstream scenarios. Our comprehensive overview addresses three key research questions: the construction of cybersecurity-oriented LLMs, the application of LLMs to various cybersecurity tasks, the challenges and further research in this area.

Overview

CoChat Summary

## TL;DR

This is a systematic literature review of LLMs in cybersecurity, covering more than 300 papers, 25 LLMs, and more than 10 downstream scenarios. It concludes that LLMs are promising for cybersecurity tasks, but their own vulnerabilities, such as jailbreaks and other attacks, remain a major concern.

## Key findings

  • The review analyzes “over 300 works,” spanning “25 LLMs” and “more than 10 downstream scenarios.”
  • It identifies three main research questions: how to construct cybersecurity-oriented domain LLMs, what cybersecurity applications LLMs support, and what challenges/future research remain.
  • The paper reports that LLMs have “great potential” across tasks including threat intelligence, vulnerability detection, secure code generation, program repair, binary analysis, IT operations, anomaly detection, and LLM-assisted attacks.
  • It highlights concrete capability benchmarks and datasets for model selection, including CyberBench (60,000+), CyberMetric (10,000), SecEval (2,000+), DebugBench (4,253), NYU CTF Dataset (200), Cybench (40), and CTIBench (2,500).
  • It also reports task-specific results such as SecureFalcon achieving “up to 96%” detection accuracy, LLMAO using bidirectional adapters for fault localization, and SafeCoder improving code security by “approximately 30%.”

## Methods

  • Systematic literature review of more than 300 academic papers since 2023, organized around three research questions.
  • The review groups prior work into cybersecurity task categories and model-construction approaches, including continual pre-training (CPT), supervised fine-tuning (SFT), full-parameter fine-tuning, and parameter-efficient fine-tuning (PEFT).
  • It synthesizes studies, benchmarks, and datasets across domains such as vulnerability detection, threat intelligence, malware detection, fuzzing, program repair, secure code generation, and LLM attacks/defenses.

## Limitations

  • The paper states that the field is still in its initial stages and that many studies are case studies without comprehensive methodology, which raises concerns about scalability and reproducibility.
  • It explicitly notes that the literature lacks connectivity and in-depth analysis between studies.
  • Evident from the text: much of the paper is a broad synthesis of cited work rather than a single unified empirical study, so many conclusions are aggregated from heterogeneous papers rather than one controlled dataset or experiment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s42400-025-00361-w

Cites

405 of 405 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

59 Valid · 6 Not valid · 101 Not checked · 49 Own results and interpretations

37 more claims · 9 with their cited sources checked

Create a free account to see all 215 claims and to verify the 18 references still open.

See all claims — free account
  • It may state two separate claims as one. 100
  • Probably not a substantial claim. 136

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 156 references

Research works printed but not linked: 33

  1. Alam MT, Bhusal D, Nguyen L et al (2024) CTIBench: a benchmark for evaluating LLMs in cyber threat intelligence. In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track
  2. Asmita, Oliinyk Y, Scott M et al (2024) Fuzzing busybox: leveraging LLM and crash reuse for embedded bug unearthing. In: Balzarotti D, Xu W (eds) 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association, https://www.usenix.org/conference/usenixsecurity24/presentation/asmita
  3. Barnum S (2012) Standardizing cyber threat intelligence information with the structured threat information expression (stix). Mitre Corp 11:1–22
  4. Bozkurt A, Sharma RC (2023) Generative ai and prompt engineering: The art of whispering to let the genie out of the algorithmic world. Asian Journal of Distance Education 18(2):i–vii
  5. Brown T, Mann B, Ryder N et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
  6. Carlini N, Tramer F, Wallace E et al (2021) Extracting training data from large language models. In: USENIX Security, pp 2633–2650
  7. Carlini N, Jagielski M, Zhang C et al (2022) The privacy onion effect: memorization is relative. In: NeurIPS, http://papers.nips.cc/paper_files/paper/2022/hash/564b5f8289ba846ebc498417e834c253-Abstract-Conference.html
  8. Chiang WL, Li Z, Lin Z et al (2023) Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna lmsys org. 2(3):6. Accessed 14 April 2023
  9. Dettmers T, Pagnoni A, Holtzman A et al (2024) Qlora: efficient finetuning of quantized llms. Adv Neural Inf Process Syst. 36
  10. Elhage N, Nanda N, Olsson C et al (2021) A mathematical framework for transformer circuits. Trans Circuits Thread 1(1):12
  11. Fang C, Miao N, Srivastav S et al (2024a) Large language models for code analysis: Do llms really do their job? In: Balzarotti D, Xu W (eds) 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association, https://www.usenix.org/conference/usenixsecurity24/presentation/fang
  12. Guo H, Yang J, Liu J et al (2024a) OWL: A large language model for IT operations. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, https://openreview.net/forum?id=SZOQ9RKYJu
  13. He J, Vero M, Krasnopolska G et al (2024) Instruction tuning for secure code generation. In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, https://openreview.net/forum?id=MgTzMaYHvG
  14. Hu EJ, Shen Y, Wallis P et al (2022) Lora: Low-rank adaptation of large language models. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, https://openreview.net/forum?id=nZeVKeeFYf9
  15. Huang L, Yu W, Ma W et al (2023a) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans Inf Syst
  16. Huang Y, Sun L, Wang H et al (2024) Trustllm: Trustworthiness in large language models. In: Forty-first International Conference on Machine Learning, ICML 2024,Vienna, Austria, July 21-27, 2024
  17. Huben R, Cunningham H, Smith LR et al (2023) Sparse autoencoders find highly interpretable features in language models. In: The Twelfth International Conference on Learning Representations
  18. Li R, Allal LB, Zi Y et al (2023d) Starcoder: may the source be with you! Trans Mach Learn Res 2023. https://openreview.net/forum?id=KoFOg41haE
  19. Li N, Pan A, Gopal A et al (2024c) The WMDP benchmark: Measuring and reducing malicious use with unlearning. In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net
  20. Lin Z, Cui J, Liao X et al (2024) Malla: Demystifying real-world large language model integrated malicious services. In: Balzarotti D, Xu W (eds) 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association, https://www.usenix.org/conference/usenixsecurity24/presentation/lin-zilong
  21. Liu Z, Shi J, Buford JF (2024f) Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. In: AAAI 2024 Workshop on Artificial Intelligence for Cyber Security
  22. Luo Z, Xu C, Zhao P et al (2024) Wizardcoder: Empowering code large language models with evol-instruct. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, https://openreview.net/forum?id=UnUwSIgK5W
  23. Mijwil M, Aljanabi M et al (2023) Towards artificial intelligence-based cybersecurity: the practices and chatgpt generated ways to combat cybercrime. Iraqi J Comput Sci Math 4(1):65–70
  24. Nijkamp E, Pang B, Hayashi H et al (2023b) Codegen: An open large language model for code with multi-turn program synthesis. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, https://openreview.net/forum?id=iaYcJKpY2B_
  25. Qin Y, Liang S, Ye Y et al (2024) Toolllm: Facilitating large language models to master 16000+ real-world apis. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, https://openreview.net/forum?id=dHng2O0Jjr
  26. Radford A, Narasimhan K (2018) Improving language understanding by generative pre-training. https://api.semanticscholar.org/CorpusID:49313245
  27. Sandoval G, Pearce H, Nys T et al (2023) Lost at c: A user study on the security implications of large language model code assistants. In: 32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, pp 2205–2222, https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval
  28. Sekhari A, Acharya J, Kamath G et al (2021) Remember what you want to forget: Algorithms for machine unlearning. In: NeurIPS, pp 18075–18086
  29. Tseng P, Yeh Z, Dai X et al (2024b) Using llms to automate threat intelligence analysis workflows in security operation centers. Preprint at arXiv:2407.13093 · arXiv
  30. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Adv Neural Inf Process Syst. 30
  31. Wang J, Hu X, Hou W et al (2024c) On the robustness of chatgpt: an adversarial and out-of-distribution perspective. IEEE Data Eng Bull 47(1):48–62
  32. Zhu S, Zhang R, An B et al (2024) Autodan: interpretable gradient-based adversarial attacks on large language models. In: First Conference on Language Modeling
  33. Zoph B, Raffel C, Schuurmans D et al (2022) Emergent abilities of large language models. TMLR

Cited by

See all 383

Metadata

Authors
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D.
Affiliations
Chinese Academy of Sciences (CN)
Corresponding author
  • Wen, Hui
  • Zhu, Hongsong
Venue
Cybersecurity
Published
2025-02-05
Volume
8
Issue
1
ISSN
2523-3246 (ISSN-L)
OpenAlex
W4407163436
Cited by
383
References
156
Citations by year
2026: 213 · 2025: 144
FWCI
20.46
Type
journal article
Language
en
Open access
Licence
Locations
  • Cybersecurity (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
Chinese Academy of Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When LLMs meet cybersecurity: a systematic literature review. https://app.cochat.ai/papers/10.1186/s42400-025-00361-w
The paper
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025). When LLMs meet cybersecurity: a systematic literature review. Cybersecurity. https://doi.org/10.1186/s42400-025-00361-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.