Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Automatic Exploit Generation (AEG) plays an important role in proactive assessment of software threats by identifying vulnerabilities and constructing functional payloads. Existing Large Language Model (LLM)-based methods, however, often struggle to reason about complex exploit logic and to perform runtime introspection, leaving a gap between static vulnerability analysis and dynamic memory behavior. We present PwnAgent, an LLM-driven multi-agent framework for end-to-end exploit generation that combines offensive domain knowledge with active runtime introspection. PwnAgent uses a hierarchical knowledge base for multi-stage exploit reasoning and a feedback-driven self-correction engine to calibrate dynamic memory parameters during execution. Because broad Capture The Flag (CTF) benchmarks offer limited binary-exploitation depth and pwn-specific evaluation must balance reproducibility, difficulty progression, and exploit diversity, we construct a 66-task pwn benchmark from public CTF-style challenges. The benchmark is primarily composed of Linux x86/x86-64 ELF binaries and stack-oriented tasks, with smaller format-string, heap, integer-overflow, ARM, and MIPS subsets used as limited probes beyond the dominant setting. Under the same recent Kimi-K2.6 backend, PwnAgent achieves a 62.12% end-to-end success rate, compared with 31.82% for the evaluated PwnGPT baseline, a 30.30 percentage-point gain. These paired results indicate that structured knowledge guidance, execution-grounded measurement, and feedback repair improve LLM-based exploit generation in the evaluated setting, while the absolute success rate shows that fully autonomous exploitation remains challenging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2×
“To reduce the logical errors common in direct code generation, CoT prompting makes intermediate reasoning steps explicit and helps maintain precision in complex tasks (Wei et al. 2022; Meng et al. 2024).”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087 · cited 2×
“To overcome the limitations of coarse-grained feedback signals, this approach builds a reflection loop around layout-oriented feedback (Shinn et al. 2023; Chen et al. 2024).”
ChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.810 · Cited by 490
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2309.01431 · Cited by 455 · cited 3×
“To overcome the limitations of coarse-grained feedback signals, this approach builds a reflection loop around layout-oriented feedback (Shinn et al. 2023; Chen et al. 2024).”
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454
Large Language Model guided Protocol Fuzzing
Meng, R., Mirchev, M., Böhme, M., & Roychoudhury, A. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24556 · Cited by 246 · cited 2×
“To reduce the logical errors common in direct code generation, CoT prompting makes intermediate reasoning steps explicit and helps maintain precision in complex tasks (Wei et al. 2022; Meng et al. 2024).”
Fuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Measuring Faithfulness in Chain-of-Thought Reasoning
Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13702 · Cited by 33 · cited 1×
“However, prior work has shown that generated CoT explanations are not necessarily faithful representations of a model’s internal decision process (Turpin et al. 2023; Lanham et al. 2023).”
Research works printed but not linked: 32
- Avgerinos T, Cha SK, Hao BLT, Brumley D (2011) AEG: Automatic exploit generation. In: Proceedings of the Network and Distributed System Security Symposium (NDSS 2011)
- Babaey V, Ravindran A (2025) Genxss: An ai-driven framework for automated detection of xss attacks in wafs. SoutheastCon 2025. IEEE, pp 1519–1524 · DOI
- Bao T, Wang R, Shoshitaishvili Y, Brumley D (2017) Your exploit is mine: Automatic shellcode transplant for remote exploits. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 824–839 https://doi.org/10.1109/SP.2017.67 · DOI
- Brumley D, Poosankam P, Song D, Zheng J (2008) Automatic patch-based exploit generation is possible: Techniques and implications. In: 2008 IEEE Symposium on Security and Privacy (sp 2008), pp. 143–157 https://doi.org/10.1109/SP.2008.17 · DOI · Paper page
- Cha SK, Avgerinos T, Rebert A, Brumley D (2012) Unleashing mayhem on binary code. In: 2012 IEEE Symposium on Security and Privacy, pp. 380–394 https://doi.org/10.1109/SP.2012.31 · DOI · Paper page
- Chen W, Zou X, Li G, Qian Z (2020)Koobe: towards facilitating exploit generation of kernel out-of-bounds write vulnerabilities. Proceedings of the 29th USENIX Conference on Security Symposium. SEC’20. USENIX Association. USA,
- Chen X, Lin M, Schärli N, Zhou D (2024) Teaching large language models to self-debug. International Conference on Learning Representations.(ICLR 2024)
- Deng G, Liu Y, Mayoral-Vilches V, Liu P, Li Y, Xu Y, Zhang T, Liu Y, Pinzger M, Rass S (2024) Pentestgpt: evaluating and harnessing large language models for automated penetration testing. In: Proceedings of the 33rd USENIX Conference on Security Symposium. SEC ’24. USENIX Association, USA
- Garmany B, Stoffel M, Gawlik R, Koppe P, Blazytko T, Holz T (2018) Towards automated generation of exploitation primitives for web browsers. In: Proceedings of the 34th Annual Computer Security Applications Conference. ACSAC ’18, pp. 300–312. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3274694.3274723 · doi:10.1145/3274694.3274723
- Heelan S, Melham T, Kroening D (2019) Gollum: Modular and greybox exploit generation for heap overflows in interpreters. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. CCS ’19, pp. 1689–1706. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3319535.3354224 · DOI
- Hu H, Chua ZL, Adrian S, Saxena P, Liang Z (2015) Automatic generation of data-oriented exploits. In: Proceedings of the 24th USENIX Conference on Security Symposium. SEC’15, pp. 177–192. USENIX Association, USA
- Huang S-K, Huang M-H, Huang P-Y, Lai C-W, Lu H-L, Leong W-M (2012) Crax: Software crash analysis for automatic exploit generation by modeling attacks as symbolic continuations. In: Proceedings of the 2012 IEEE Sixth International Conference on Software Security and Reliability. SERE ’12, pp. 78–87. IEEE Computer Society, USA https://doi.org/10.1109/SERE.2012.20 · doi:10.1109/sere.2012.20
- Ispoglou KK, AlBassam B, Jaeger T, Payer M (2018) Block oriented programming: Automating data-only attacks. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. CCS ’18, pp. 1868–1882. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3243734.3243739 · DOI
- Kaniewski S, Schmidt F, Enzweiler M, Menth M, Heer T (2025) A systematic literature review on detecting software vulnerabilities with large language models. arXiv preprint arXiv:2507.22659https://doi.org/10.48550/arXiv.2507.22659 · arXiv · doi:10.48550/arxiv.2507.22659
- Khan S (2024) Ll-xss: End-to-end generative model-based xss payload creation. In: 2024 21st Learning and Technology Conference (L&T), pp. 121–126 https://doi.org/10.1109/LT60077.2024.10469151 · DOI
- Klischies D, Mackensen P, Moonsamy V (2025) Vulnerability where art thou? an investigation of vulnerability management in android smartphone chipsets. NDSS Symposium. https://doi.org/10.14722/ndss.2025.241161 · doi:10.14722/ndss.2025.241161
- Kurmus A, Mambretti A, Sorniotti A, Lenders V, Pfammatter D, Tellenbach B (2025) SoK: Automating kernel vulnerability discovery and exploit generation. 19th USENIX WOOT Conference on Offensive Technologies (WOOT ’25). pp 283–302
- Liu J, An H, Li J, Liang H (2022) Detecting exploit primitives automatically for heap vulnerabilities on binary programs. https://doi.org/10.48550/arXiv.2212.13990 · arXiv · doi:10.48550/arxiv.2212.13990
- Peng W, Ye L, Du X, Zhang H, Zhan D, Zhang Y, Guo Y, Zhang C (2025) PwnGPT: Automatic exploit generation based on large language models. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11481–11494. Association for Computational Linguistics, Vienna, Austria https://doi.org/10.18653/v1/2025.acl-long.562 · DOI
- Ray S, Pan R, Gu Z, Du K, Feng S, Ananthanarayanan G, Netravali R, Jiang J (2025) Metis: fast quality-aware rag systems with configuration adaptation. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. pp 606–622 · DOI
- Shao M, Xi H, Rani N, Udeshi M, Putrevu VSC, Milner K, Dolan-Gavitt B, Shukla SK, Krishnamurthy P, Khorrami F, Karri R, Shafique M (2025) CRAKEN: Cybersecurity LLM agent with knowledge-based execution. arXiv preprint arXiv:2505.17107https://doi.org/10.48550/arXiv.2505.17107 · DOI
- Turpin M, Michael J, Perez E, Bowman S (2023) Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Adv Neural Inf Process Syst 36:74952–74965 · doi:10.52202/075280-3275 · Paper page
- Wang Y, Zhang C, Xiang X, Zhao Z, Li W, Gong X, Liu B, Chen K, Zou W (2018) Revery: From proof-of-concept to exploitable. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. CCS ’18, pp. 1914–1927. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3243734.3243847 · doi:10.1145/3243734.3243847
- Wang D, Zhou G, Chen L, Li D, Miao Y (2024) Prophetfuzz: Fully automated prediction and fuzzing of high-risk option combinations with only documentation via large language model. In: Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. CCS ’24, pp. 735–749. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3658644.3690231 · DOI
- Wu W, Chen Y, Xu J, Xing X, Gong X, Zou W (2018) Fuze: towards facilitating exploit generation for kernel use-after-free vulnerabilities. In: Proceedings of the 27th USENIX Conference on Security Symposium. SEC’18, pp. 781–797. USENIX Association, USA
- Wu Q, Bansal G, Zhang J, Wu Y, Li B, Zhu E, Jiang L, Zhang X, Zhang S, Liu J et al.: (2024) Autogen: Enabling next-gen llm applications via multi-agent conversations. In: First Conference on Language Modeling
- Xiao Z, Wang Q, Li Y, Chen S (2025) Prompt to pwn: Automated exploit generation for smart contracts. arXiv preprint arXiv:2508.01371https://doi.org/10.48550/arXiv.2508.01371 · arXiv · doi:10.48550/arxiv.2508.01371
- Xu J, Stokes JW, McDonald G, Bai X, Marshall D, Wang S, Swaminathan A, Li Z (2024) Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv preprint arXiv:2403.01038https://doi.org/10.48550/arXiv.2403.01038 · DOI
- Xu C, Guan S, Greene D, Kechadi M (2024) Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244 · arXiv
- Yang G, Zhou Y, Chen X, Zhang X, Han T, Chen T (2023) Exploitgen: Template-augmented exploit code generation based on codebert. J Syst Softw 197:111577. https://doi.org/10.1016/j.jss.2022.111577 · DOI
- Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K, Cao Y (2023) ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR)
- Zhu Y, Kellermann A, Gupta A, Li P, Fang R, Bindu R, Kang D (2024) Teams of LLM agents can exploit zero-day vulnerabilities. https://doi.org/10.48550/arXiv.2406.01637 · DOI
Metadata
- Authors
- Wei, C., Geng, Y., Wang, Y., Wu, Q., Huang, J., Wu, Q., & Wei, Q.
- Venue
- Cybersecurity
- Published
- 2026-09-28
- Volume
- 9
- Issue
- 1
- ISSN
- 2523-3246
- Cited by
- 0
- References
- 16
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation. https://app.cochat.ai/papers/10.1186/s42400-026-00649-5
- The paper
- Wei, C., Geng, Y., Wang, Y., Wu, Q., Huang, J., Wu, Q., & Wei, Q. (2026). Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation. Cybersecurity. https://doi.org/10.1186/s42400-026-00649-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.