Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Large language model (LLM) agents increasingly automate offensive-security tasks, but success on isolated challenges does not establish reliable long-horizon autonomy. We study this distinction in authorized capture-the-flag (CTF) and wargame environments and introduce a planner-directed architecture in which a persistent non-executing planner delegates on demand to transient, context-isolated specialists with retrievable skills, persistent execution, compact memory, guardrails, and configurable model escalation. On all 200 challenges of the NYU CTF Bench test split, the system obtains 41 accepted flags (20.5%) and 29 leak-audited genuine solves (14.5%), against 4 (2.0%) and 3 (1.5%) for a stateful single-agent baseline. Both start from gpt-4o-mini, but only the planner-directed system could escalate to gpt-5-mini, which it did on 199 of 200 challenges and for 72.5% of spend. A controlled architecture-by-escalation experiment over a pre-registered subset of 80 challenges, crossing a resource-matched monolithic agent with the planner under matched budgets, was run in two independent formal passes. M0 solves 4/80 then 3/80, M1 22/80 then 19/80, P0 4/80 then 3/80, and P1 14/80 then 15/80. Pooling the two formal passes with challenge as the repeated unit, a clustered GEE shows a strong escalation effect (OR 8.79, 95% CI 3.39−22.81, $$p = 8\times 10^{-6}$$ p = 8 × 10 - 6 ), no architecture main effect (OR 1.17, $$p = 0.34$$ p = 0.34 ), and a significant architecture-by-escalation interaction (OR 0.53, $$p = 0.005$$ p = 0.005 ). Under the spend-constrained 340-action ceiling, the coordination cost of specialist delegation, context restart, and evidence folding offsets any additional solve-rate advantage from the planner-directed structure. On OverTheWire Bandit, the planner-directed system reaches 30/33 standalone and 24/33 sequentially, compared with 27/33 and 23/33 for the strongest single-agent mini backbone.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., Xi, H., Milner, K., Chen, B., Yin, M., Garg, S., Krishnamurthy, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.05590 · Cited by 18
Research works printed but not linked: 14
- Abramovich, T., Udeshi, M., Shao, M., Lieret, K., Xi, H., Milner, K., Jancheska, S., Yang, J., Jimenez, C.E., Khorrami, F., Krishnamurthy, P., Dolan-Gavitt, B., Shafique, M., Narasimhan, K.R., Karri, R., Press, O.: EnIGMA: interactive tools substantially assist LM agents in finding security vulnerabilities. In: Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., Zhu, J. (eds.) Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 267, pp. 246–355. PMLR (2025)
- Amine, A., Zayat, A., Saab, S.: Stateful llm agents for autonomous capture-the-flag solving. In: 2026 4th Cognitive Models and Artificial Intelligence Conference (AICCONF), pp. 1–8. IEEE (2026). https://doi.org/10.1109/AICCONF69182.2026.11600706 · DOI
- Bakker, I., Hastings, J.: Autonomous penetration testing: Solving capture-the-flag challenges with llms. In: 2025 Cyber Awareness and Research Symposium (CARS), pp. 1–6. IEEE (2025). https://doi.org/10.1109/CARS67163.2025.11337515 · DOI
- Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: $$\{$$PentestGPT$$\}$$: evaluating and harnessing large language models for automated penetration testing. In: 33rd USENIX Security Symposium (USENIX Security 24), pp. 847–864 (2024)
- Erdődi, L., Zennaro, F.M.: The agent web model: modeling web hacking for reinforcement learning. Int. J. Inf. Secur. 21(2), 293–309 (2022). https://doi.org/10.1007/s10207-021-00554-7 · DOI
- Hilario, E., Azam, S., Sundaram, J., Imran Mohammed, K., Shanmugam, B.: Generative ai for pentesting: the good, the bad, the ugly. Int. J. Inf. Secur. 23(3), 2075–2097 (2024). https://doi.org/10.1007/s10207-024-00835-x · DOI · Paper page
- Lee, D., Bae, G.e., Yun, I.: CTFusion: a ctf-based benchmark for llm agent evaluation. In: Proceedings of the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD), ICML 2026 (2026)
- Merves, T.H., Conaway, M.H., Escobar, J.M., Otal, H.T., Tatar, U.: Systematic capability benchmarking of frontier large language models for offensive cyber tasks. In: 2026 Systems and Information Engineering Design Symposium (SIEDS), pp. 1–6. IEEE (2026). https://doi.org/10.1109/SIEDS69358.2026.11540254 · DOI
- Shao, M., Xi, H., Rani, N., Udeshi, M., Putrevu, V.S.C., Milner, K., Dolan-Gavitt, B., Shukla, S.K., Krishnamurthy, P., Khorrami, F., et al.: Craken: cybersecurity llm agent with knowledge-based execution. arXiv preprint arXiv:2505.17107 (2025). https://doi.org/10.48550/arXiv.2505.17107 · DOI
- Sommervoll, Å.Å., Erdődi, L., Zennaro, F.M.: Simulating all archetypes of sql injection vulnerability exploitation using reinforcement learning agents. Int. J. Inf. Secur. 23(1), 225–246 (2024). https://doi.org/10.1007/s10207-023-00738-3 · DOI
- Udeshi, M., Shao, M., Xi, H., Rani, N., Milner, K., Putrevu, V.S.C., Dolan-Gavitt, B., Shukla, S.K., Krishnamurthy, P., Khorrami, F., et al.: D-CIPHER: dynamic collaborative intelligent agents with planning and heterogeneous execution for enhanced reasoning in offensive security. arXiv preprint arXiv:2502.10931 (2025). https://doi.org/10.48550/arXiv.2502.10931 · DOI
- Wen, S.F., Shukla, A., Katt, B.: Artificial intelligence for system security assurance: a systematic literature review. Int. J. Inf. Secur. 24(1), 43 (2025). https://doi.org/10.1007/s10207-024-00959-0 · DOI
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: International Conference on Learning Representations (2023). https://openreview.net/forum?id=WE_vluYUL-X
- Zhu, Y., Kellermann, A., Gupta, A., Li, P., Fang, R., Bindu, R., Kang, D.: Teams of llm agents can exploit zero-day vulnerabilities. In: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23–35 (2026). https://doi.org/10.18653/v1/2026.eacl-long.2 · DOI · Paper page
Metadata
- Authors
- Amine, A., & Saab, S.
- Venue
- International Journal of Information Security
- Published
- 2026-11-05
- Volume
- 25
- Issue
- 6
- ISSN
- 1615-5270
- Cited by
- 0
- References
- 2
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing. https://app.cochat.ai/papers/10.1007/s10207-026-01350-x
- The paper
- Amine, A., & Saab, S. (2026). Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing. International Journal of Information Security. https://doi.org/10.1007/s10207-026-01350-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.