NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Open accessAbstract
Large Language Models (LLMs) are being deployed across various domains today. However, their capacity to solve Capture the Flag (CTF) challenges in cybersecurity has not been thoroughly evaluated. To address this, we develop a novel method to assess LLMs in solving CTF challenges by creating a scalable, open-source benchmark database specifically designed for these applications. This database includes metadata for LLM testing and adaptive learning, compiling a diverse range of CTF challenges from popular competitions. Utilizing the advanced function calling capabilities of LLMs, we build a fully automated system with an enhanced workflow and support for external tool calls. Our benchmark dataset and automated framework allow us to evaluate the performance of five LLMs, encompassing both black-box and open-source models. This work lays the foundation for future research into improving the efficiency of LLMs in interactive cybersecurity tasks and automated task planning. By providing a specialized benchmark, our project offers an ideal platform for developing, testing, and refining LLM-based approaches to vulnerability detection and resolution. Evaluating LLMs on these challenges and comparing with human performance yields insights into their potential for AI-driven cybersecurity solutions to perform real-world threat management. We make our benchmark dataset open source to public https://github.com/NYU-LLM-CTF/NYU_CTF_Bench along with our playground automated framework https://github.com/NYU-LLM-CTF/llm_ctf_automation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
NYU CTF Dataset and Cybench are used to assess LLMs capacity to solve Capture the Flag (CTF) challenges in cybersecurity, aiming to improve the efficiency of LLMs in interactive cybersecurity tasks and automated task planning.
“NYU CTF Dataset(Shao etal. 2024) and Cybench (Zhang et al. 2024a) are used to assess LLMs capacity to solve Capture the Flag (CTF) challenges in cybersecurity, aiming to improve the efficiency of LLMs in interactive cybersecurity tasks and automated task planning.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Shao et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 30
- J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021. URL https: //arxiv.org/abs/2108.07732.
- T. J. Burns, S. C. Rios, T. K. Jordan, Q. Gu, and T. Underwood. Analysis and exercises for engaging beginners in online CTF competitions for security education. In 2017 USENIX Workshop on Advances in Security Education (ASE 17), Vancouver, BC, Aug. 2017. USENIX Association. URL https://www.usenix.org/conference/ase17/workshop-program/ presentation/burns.
- G. Chan. Ai employment decision-making: integrating the equal opportunity merit principle and explainable ai. AI & SOCIETY, 07 2022. doi: 10.1007/s00146-022-01532-w. · DOI
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
- A. Cheok, A. Sreekumar, C. Lei, and L. Thang. Capture the flag: mixed-reality social gaming with smart phones. IEEE Pervasive Computing, 5(2):62-69, 2006. doi: 10.1109/MPRV.2006.25. · DOI
- R. G. Chicone and S. Ferebee. A comparison study of two cybersecurity learning systems: facebook's open-source capture the flag and ctfd. Issues in Information Systems, 21(1):202-212, 2020.
- G. Costa, M. Lualdi, M. Ribaudo, and A. Valenza. A nerd dogma: Introducing ctf to non-expert audience. In Proceedings of the 21st Annual Conference on Information Technology Education, SIGITE '20, page 413-418, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450370455. doi: 10.1145/3368308.3415405. URL https://doi.org/10.1145/ 3368308.3415405. · DOI
- G. Deng, Y. Liu, V. Mayoral-Vilches23, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, and S. Rass. Pentestgpt: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium. USENIX, 2024.
- F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017. · arXiv
- R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018. · arXiv
- A. H. A. Hanafi, H. Rokman, A. D. Ibrahim, Z.-A. Ibrahim, M. N. A. Zawawi, and F. A. Rahim. A ctf-based approach in cyber security education for secondary school students. Electronic Journal of Computer Science and Information Technology, 7(1), 2021.
- D. Hendrycks, M. Mazeika, A. Zou, and D. Song. Aligning ai with shared human values, 2020. URL https://arxiv.org/pdf/2009.03300. · arXiv
- Z. Kaplan, N. Zhang, and S. V. Cole. A capture the flag (ctf) platform and exercises for an intro to computer security class. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2, pages 597-598, 2022.
- S. Karagiannis, E. Maragkos-Belmpas, and E. Magkos. An analysis and evaluation of open source capture the flag platforms as cybersecurity e-learning tools. In L. Drevin, S. Von Solms, and M. Theocharidou, editors, Information Security Education. Information Security in Action, pages 61-77, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59291-2.
- S. Karagiannis, E. Magkos, G. Chalavazis, and M. N. Nikiforos. Analysis and evaluation of capture the flag challenges in secure mobile application development. International Journal on Integrating Technology in Education, 11:19-35, 06 2022. doi: 10.5121/ijite.2022.11202. · DOI
- S. Kucek and M. Leitner. An empirical survey of functions and configurations of open-source capture the flag (ctf) environments. Journal of Network and Computer Applications, 151: 102470, 2020. ISSN 1084-8045. doi: https://doi.org/10.1016/j.jnca.2019.102470. URL https://www.sciencedirect.com/science/article/pii/S1084804519303303. · DOI
- W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
- K. Leune and S. J. Petrilli Jr. Using capture-the-flag to enhance the effectiveness of cybersecurity education. In Proceedings of the 18th annual conference on information technology education, pages 47-52, 2017.
- L. McDaniel, E. Talvi, and B. Hay. Capture the flag as cyber security introduction. In 2016 Hawaii International Conference on System Sciences (hicss), pages 5479-5486. IEEE, 2016.
- H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt. Examining zero-shot vulnerability repair with large language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2339-2356, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: 10.1109/SP46215.2023.10179420. URL https://doi.ieeecomputersociety.org/10. 1109/SP46215.2023.10179420. · DOI
- T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL https://arxiv.org/abs/2302.04761. · arXiv
- M. Shao, B. Chen, S. Jancheska, B. Dolan-Gavitt, S. Garg, R. Karri, and M. Shafique. An empirical evaluation of llms for solving offensive security challenges, 2024. URL https: //arxiv.org/abs/2402.11814.
- Y. Shoshitaishvili, R. Wang, C. Hauser, C. Kruegel, G. Vigna, and M. Wiesner. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. In 2016 IEEE Symposium on Security and Privacy (SP), pages 138-157. IEEE, 2016. doi: 10.1109/SP.2016.15. URL https://doi.org/10.1109/SP.2016.15. · DOI
- J. T, J. A, and N. Nelmiawati. Analysis of cyber security knowledge and skills for capture the flag competition. JURNAL INTEGRASI, 14:14-22, 04 2022. doi: 10.30871/ji.v14i1.3986. · DOI
- W. Tann, Y. Liu, J. H. Sim, C. M. Seah, and E.-C. Chang. Using large language models for cybersecurity capture-the-flag challenges and certification questions, 2023. URL https: //arxiv.org/abs/2308.10443.
- J. Vykopal, V. Švábenský, and E.-C. Chang. Benefits and pitfalls of using capture the flag games in university courses. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE '20, page 752-758, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367936. doi: 10.1145/3328778.3366893. URL https://doi.org/10.1145/3328778.3366893. · DOI
- Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks), 2024. URL https://arxiv.org/abs/2406.01574. · arXiv
- X. Wu, R. Duan, and J. Ni. Unveiling security, privacy, and ethical concerns of chatgpt. Journal of Information and Intelligence, 2(2):102-115, 2024. ISSN 2949-7159. doi: https: //doi.org/10.1016/j.jiixd.2023.10.007. URL https://www.sciencedirect.com/science/ article/pii/S2949715923000707. · DOI
- J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. corr, abs/2306.14898, 2023d. doi: 10.48550. arXiv preprint ARXIV.2306.14898, 2023. · arXiv
- J. Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop @ NeurIPS'23, 2023. URL https://openreview.net/forum?id=KOZwk7BFc3.
Cited by
Sorted by citations
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
- Authors
- Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., Xi, H., Milner, K., Chen, B., Yin, M., Garg, S., Krishnamurthy, P., Khorrami, F., Karri, R., & Shafique, M.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 57472-57498
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2406.05590
- OpenAlex
- W7103749693
- Cited by
- 18
- References
- 0
- Citations by year
- 2026: 16 · 2025: 2
- FWCI
- 5.20
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2016650)
- New York University Abu Dhabi (CG010, G1104)
- Tamkeen (CG010, G1104)
- Office of Naval Research (N00014-22, N00014)
- Army Research Office (W911NF-22-1-0028, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- large language models, offensive security, benchmark dataset, open-source benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
