CoChat

Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing

Amine, Amir, Saab, Samer · 2026 · International Journal of Information Security
Cited by 0

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

Large language model (LLM) agents increasingly automate offensive-security tasks, but success on isolated challenges does not establish reliable long-horizon autonomy. We study this distinction in authorized capture-the-flag (CTF) and wargame environments and introduce a planner-directed architecture in which a persistent non-executing planner delegates on demand to transient, context-isolated specialists with retrievable skills, persistent execution, compact memory, guardrails, and configurable model escalation. On all 200 challenges of the NYU CTF Bench test split, the system obtains 41 accepted flags (20.5%) and 29 leak-audited genuine solves (14.5%), against 4 (2.0%) and 3 (1.5%) for a stateful single-agent baseline. Both start from gpt-4o-mini, but only the planner-directed system could escalate to gpt-5-mini, which it did on 199 of 200 challenges and for 72.5% of spend. A controlled architecture-by-escalation experiment over a pre-registered subset of 80 challenges, crossing a resource-matched monolithic agent with the planner under matched budgets, was run in two independent formal passes. M0 solves 4/80 then 3/80, M1 22/80 then 19/80, P0 4/80 then 3/80, and P1 14/80 then 15/80. Pooling the two formal passes with challenge as the repeated unit, a clustered GEE shows a strong escalation effect (OR 8.79, 95% CI 3.39−22.81, $$p = 8\times 10^{-6}$$ p = 8 × 10 - 6 ), no architecture main effect (OR 1.17, $$p = 0.34$$ p = 0.34 ), and a significant architecture-by-escalation interaction (OR 0.53, $$p = 0.005$$ p = 0.005 ). Under the spend-constrained 340-action ceiling, the coordination cost of specialist delegation, context restart, and evidence folding offsets any additional solve-rate advantage from the planner-directed structure. On OverTheWire Bandit, the planner-directed system reaches 30/33 standalone and 24/33 sequentially, compared with 27/33 and 23/33 for the strongest single-agent mini backbone.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10207-026-01350-x

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 14

  1. Abramovich, T., Udeshi, M., Shao, M., Lieret, K., Xi, H., Milner, K., Jancheska, S., Yang, J., Jimenez, C.E., Khorrami, F., Krishnamurthy, P., Dolan-Gavitt, B., Shafique, M., Narasimhan, K.R., Karri, R., Press, O.: EnIGMA: interactive tools substantially assist LM agents in finding security vulnerabilities. In: Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., Zhu, J. (eds.) Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 267, pp. 246–355. PMLR (2025)
  2. Amine, A., Zayat, A., Saab, S.: Stateful llm agents for autonomous capture-the-flag solving. In: 2026 4th Cognitive Models and Artificial Intelligence Conference (AICCONF), pp. 1–8. IEEE (2026). https://doi.org/10.1109/AICCONF69182.2026.11600706 · DOI
  3. Bakker, I., Hastings, J.: Autonomous penetration testing: Solving capture-the-flag challenges with llms. In: 2025 Cyber Awareness and Research Symposium (CARS), pp. 1–6. IEEE (2025). https://doi.org/10.1109/CARS67163.2025.11337515 · DOI
  4. Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: $$\{$$PentestGPT$$\}$$: evaluating and harnessing large language models for automated penetration testing. In: 33rd USENIX Security Symposium (USENIX Security 24), pp. 847–864 (2024)
  5. Erdődi, L., Zennaro, F.M.: The agent web model: modeling web hacking for reinforcement learning. Int. J. Inf. Secur. 21(2), 293–309 (2022). https://doi.org/10.1007/s10207-021-00554-7 · DOI
  6. Hilario, E., Azam, S., Sundaram, J., Imran Mohammed, K., Shanmugam, B.: Generative ai for pentesting: the good, the bad, the ugly. Int. J. Inf. Secur. 23(3), 2075–2097 (2024). https://doi.org/10.1007/s10207-024-00835-x · DOI · Paper page
  7. Lee, D., Bae, G.e., Yun, I.: CTFusion: a ctf-based benchmark for llm agent evaluation. In: Proceedings of the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD), ICML 2026 (2026)
  8. Merves, T.H., Conaway, M.H., Escobar, J.M., Otal, H.T., Tatar, U.: Systematic capability benchmarking of frontier large language models for offensive cyber tasks. In: 2026 Systems and Information Engineering Design Symposium (SIEDS), pp. 1–6. IEEE (2026). https://doi.org/10.1109/SIEDS69358.2026.11540254 · DOI
  9. Shao, M., Xi, H., Rani, N., Udeshi, M., Putrevu, V.S.C., Milner, K., Dolan-Gavitt, B., Shukla, S.K., Krishnamurthy, P., Khorrami, F., et al.: Craken: cybersecurity llm agent with knowledge-based execution. arXiv preprint arXiv:2505.17107 (2025). https://doi.org/10.48550/arXiv.2505.17107 · DOI
  10. Sommervoll, Å.Å., Erdődi, L., Zennaro, F.M.: Simulating all archetypes of sql injection vulnerability exploitation using reinforcement learning agents. Int. J. Inf. Secur. 23(1), 225–246 (2024). https://doi.org/10.1007/s10207-023-00738-3 · DOI
  11. Udeshi, M., Shao, M., Xi, H., Rani, N., Milner, K., Putrevu, V.S.C., Dolan-Gavitt, B., Shukla, S.K., Krishnamurthy, P., Khorrami, F., et al.: D-CIPHER: dynamic collaborative intelligent agents with planning and heterogeneous execution for enhanced reasoning in offensive security. arXiv preprint arXiv:2502.10931 (2025). https://doi.org/10.48550/arXiv.2502.10931 · DOI
  12. Wen, S.F., Shukla, A., Katt, B.: Artificial intelligence for system security assurance: a systematic literature review. Int. J. Inf. Secur. 24(1), 43 (2025). https://doi.org/10.1007/s10207-024-00959-0 · DOI
  13. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: International Conference on Learning Representations (2023). https://openreview.net/forum?id=WE_vluYUL-X
  14. Zhu, Y., Kellermann, A., Gupta, A., Li, P., Fang, R., Bindu, R., Kang, D.: Teams of llm agents can exploit zero-day vulnerabilities. In: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23–35 (2026). https://doi.org/10.18653/v1/2026.eacl-long.2 · DOI · Paper page

Metadata

Authors
Amine, A., & Saab, S.
Venue
International Journal of Information Security
Published
2026-11-05
Volume
25
Issue
6
ISSN
1615-5270
Cited by
0
References
2
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing. https://app.cochat.ai/papers/10.1007/s10207-026-01350-x
The paper
Amine, A., & Saab, S. (2026). Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing. International Journal of Information Security. https://doi.org/10.1007/s10207-026-01350-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.