LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks
Open accessAbstract
Large Language Models (LLMs) have been suggested for use in automated vulnerability repair, but benchmarks showing they can consistently identify security-related bugs are lacking. We thus develop SecLLMHolmes, a fully automated evaluation framework that performs the most detailed investigation to date on whether LLMs can reliably identify and reason about security-related bugs. We construct a set of 228 code scenarios and analyze eight of the most capable LLMs across eight different investigative dimensions using our framework. Our evaluation shows LLMs provide non-deterministic responses, incorrect and unfaithful reasoning, and perform poorly in real-world scenarios. Most importantly, our findings reveal significant non-robustness in even the most advanced models like `PaLM2' and `GPT-4': by merely changing function or variable names, or by the addition of library functions in the source code, these models can yield incorrect answers in 26% and 17% of cases, respectively. These findings demonstrate that further LLM advances are needed before LLMs can be used as general purpose security assistants.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ullah et al. (2023) point out that current LLMs fall short in automated vulnerability detection tasks and outline several limitations exhibited by current LLMs.
“They point out that current LLMs fall short in automated vulnerability detection tasks and outline several limitations exhibited by current LLMs.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Ullah et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 34
- S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, and Radford, "Evaluating Large Language Models Trained on Code," arXiv e-prints, p. arXiv:2107.03374, Jul. 2021. · arXiv
- B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, "Code Llama: Open Foundation Models for Code," arXiv e-prints, p. arXiv:2308.12950, Aug. 2023. · arXiv
- R. Li, L. Ben Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.- H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. Stillerman, S. Sankalp Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, N. Fahmy, U. Bhattacharyya, W. Yu, S. Singh, S. Luccioni, P. Villegas, M. Kunakov, F. Zhdanov, M. Romero, T. Lee, N. Timor, and J. Ding, "StarCoder: may the source be with you!" arXiv e-prints, p. arXiv:2305.06161, May 2023. · arXiv
- H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, "Ex- amining zero-shot vulnerability repair with large language models," in IEEE Symposium on Security and Privacy, 2023.
- H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "Asleep at the keyboard? assessing the security of github copilot's code contributions," in IEEE Symposium on Security and Privacy, 2022.
- N. Perry, M. Srivastava, D. Kumar, and D. Boneh, "Do users write more insecure code with ai assistants?" in Proceedings of ACM SIGSAC Conference on Computer and Communications Security, 2023.
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reason- ing in large language models," in Advances in Neural Information Processing Systems, 2022.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, 2017.
- J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, "Emer- gent abilities of large language models," Transactions on Machine Learning Research, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, and Berner, "Language models are few- shot learners," in Advances in Neural Information Processing Systems, 2020.
- F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, "Modeling and discovering vulnerabilities with code property graphs," in IEEE Sym- posium on Security and Privacy, 2014.
- Y. Mirsky, G. Macon, M. Brown, C. Yagemann, M. Pruett, E. Down- ing, S. Mertoguno, and W. Lee, "VulChecker: Graph-based vul- nerability localization in source code," in 32nd USENIX Security Symposium, 2023.
- Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, "Vuldeepecker: A deep learning-based system for vulnerability de- tection," Proceedings of Network and Distributed System Security Symposium, 2018.
- G. Lin, J. Zhang, W. Luo, L. Pan, and Y. Xiang, "Poster: Vulnera- bility discovery with function representation learning from unlabeled projects," in Proceedings of ACM SIGSAC Conference on Computer and Communications Security, 2017.
- Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, "Sysevr: A framework for using deep learning to detect software vulnerabilities," IEEE Transactions on Dependable and Secure Computing, 2022.
- D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, "UniXcoder: Unified cross-modal pre-training for code representation," in Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics, May 2022.
- H. Hanif and S. Maffeis, "Vulberta: Simplified source code pre- training for vulnerability detection," in International Joint Conference on Neural Networks (IJCNN), 2022.
- L. Phan, H. Tran, D. Le, H. Nguyen, J. Annibal, A. Peltekian, and Y. Ye, "CoTexT: Multi-task learning with code-text transformer," in Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog), 2021.
- D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, "Dos and don'ts of machine learning in computer security," in 31st USENIX Security Symposium, 2022.
- N. Risse and M. Böhme, "Limits of Machine Learning for Automatic Vulnerability Detection," arXiv e-prints, p. arXiv:2306.17193, Jun. 2023. · arXiv
- W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, "Unified pre- training for program understanding and generation," in Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
- Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, "Devign: Effective vul- nerability identification by learning comprehensive program seman- tics via graph neural networks," in Advances in Neural Information Processing Systems, 2019.
- C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, "Transformer-based language models for software vul- nerability detection," in Proceedings of the 38th Annual Computer Security Applications Conference, 2022.
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," in Advances in Neural Information Processing Systems, 2022, pp. 22 199-22 213.
- M. Nye, A. Andreassen, G. Gur-Ari, H. W. Michalewski, J. Austin, D. Bieber, D. M. Dohan, A. Lewkowycz, M. P. Bosma, D. Luan, C. Sutton, and A. Odena, "Show your work: Scratch- pads for intermediate computation with language models," 2021, https://arxiv.org/abs/2112.00114. · arXiv
- D. Votipka, R. Stevens, E. Redmiles, J. Hu, and M. Mazurek, "Hack- ers vs. testers: A comparison of software vulnerability discovery processes," in IEEE Symposium on Security and Privacy, 2018.
- D. Votipka, S. Rabin, K. Micinski, J. S. Foster, and M. L. Mazurek, "An observational investigation of reverse Engineers' processes," in 29th USENIX Security Symposium, 2020.
- R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, "Automated vulnerability detection in source code using deep representation learning," in 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018.
- A. Rahman, C. Parnin, and L. Williams, "The seven sins: Security smells in infrastructure as code scripts," in IEEE/ACM 41st Interna- tional Conference on Software Engineering (ICSE), 2019.
- K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Zhenqiang Gong, and X. Xie, "PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts," arXiv e-prints, p. arXiv:2306.04528, 2023. · arXiv
- C. Li, J. Wang, Y. Zhang, K. Zhu, W. Hou, J. Lian, F. Luo, Q. Yang, and X. Xie, "Large Language Models Understand and Can be En- hanced by Emotional Stimuli," arXiv e-prints, p. arXiv:2307.11760, 2023. · arXiv
- J. L. Fleiss, B. Levin, and M. C. Paik, Statistical methods for rates and proportions. Wiley-Interscience, 2003.
- C. Y. Lin, "ROUGE: A package for automatic evaluation of sum- maries." Association for Computational Linguistics, Jul. 2004.
- A. Creswell and M. Shanahan, "Faithful Reasoning Using Large Language Models," arXiv e-prints, p. arXiv:2208.14271, Aug. 2022. · arXiv
Cited by
Sorted by citations
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69 · see citing works
- Authors
- Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., & Stringhini, G.
- Affiliations
- Boston University (US)
- IBM (United States) (US)
- UNSW Sydney (AU)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 862-880
- ISSN
- 1081-6011 (ISSN-L)
- arXiv
- 2312.12575
- OpenAlex
- W4402264467
- Cited by
- 132
- References
- 0
- Citations by year
- 2026: 65 · 2025: 51 · 2024: 16
- FWCI
- 35.92
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- OpenBU (Boston University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2127232, 1942610)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- large language models, security vulnerabilities, vulnerability detection, automated vulnerability repair, evaluation framework, benchmarking, reasoning errors, model robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
