CoChat

Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation

Wei, Chaojie, Geng, Yangyang, Wang, Yunfeng, Wu, Qilong, Huang, Jing, Wu, Qianqiong, Wei, Qiang · 2026 · Cybersecurity
Cited by 0

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Abstract

Automatic Exploit Generation (AEG) plays an important role in proactive assessment of software threats by identifying vulnerabilities and constructing functional payloads. Existing Large Language Model (LLM)-based methods, however, often struggle to reason about complex exploit logic and to perform runtime introspection, leaving a gap between static vulnerability analysis and dynamic memory behavior. We present PwnAgent, an LLM-driven multi-agent framework for end-to-end exploit generation that combines offensive domain knowledge with active runtime introspection. PwnAgent uses a hierarchical knowledge base for multi-stage exploit reasoning and a feedback-driven self-correction engine to calibrate dynamic memory parameters during execution. Because broad Capture The Flag (CTF) benchmarks offer limited binary-exploitation depth and pwn-specific evaluation must balance reproducibility, difficulty progression, and exploit diversity, we construct a 66-task pwn benchmark from public CTF-style challenges. The benchmark is primarily composed of Linux x86/x86-64 ELF binaries and stack-oriented tasks, with smaller format-string, heap, integer-overflow, ARM, and MIPS subsets used as limited probes beyond the dominant setting. Under the same recent Kimi-K2.6 backend, PwnAgent achieves a 62.12% end-to-end success rate, compared with 31.82% for the evaluated PwnGPT baseline, a 30.30 percentage-point gain. These paired results indicate that structured knowledge guidance, execution-grounded measurement, and feedback repair improve LLM-based exploit generation in the evaluated setting, while the absolute success rate shows that fully autonomous exploitation remains challenging.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s42400-026-00649-5

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 32

  1. Avgerinos T, Cha SK, Hao BLT, Brumley D (2011) AEG: Automatic exploit generation. In: Proceedings of the Network and Distributed System Security Symposium (NDSS 2011)
  2. Babaey V, Ravindran A (2025) Genxss: An ai-driven framework for automated detection of xss attacks in wafs. SoutheastCon 2025. IEEE, pp 1519–1524 · DOI
  3. Bao T, Wang R, Shoshitaishvili Y, Brumley D (2017) Your exploit is mine: Automatic shellcode transplant for remote exploits. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 824–839 https://doi.org/10.1109/SP.2017.67 · DOI
  4. Brumley D, Poosankam P, Song D, Zheng J (2008) Automatic patch-based exploit generation is possible: Techniques and implications. In: 2008 IEEE Symposium on Security and Privacy (sp 2008), pp. 143–157 https://doi.org/10.1109/SP.2008.17 · DOI · Paper page
  5. Cha SK, Avgerinos T, Rebert A, Brumley D (2012) Unleashing mayhem on binary code. In: 2012 IEEE Symposium on Security and Privacy, pp. 380–394 https://doi.org/10.1109/SP.2012.31 · DOI · Paper page
  6. Chen W, Zou X, Li G, Qian Z (2020)Koobe: towards facilitating exploit generation of kernel out-of-bounds write vulnerabilities. Proceedings of the 29th USENIX Conference on Security Symposium. SEC’20. USENIX Association. USA,
  7. Chen X, Lin M, Schärli N, Zhou D (2024) Teaching large language models to self-debug. International Conference on Learning Representations.(ICLR 2024)
  8. Deng G, Liu Y, Mayoral-Vilches V, Liu P, Li Y, Xu Y, Zhang T, Liu Y, Pinzger M, Rass S (2024) Pentestgpt: evaluating and harnessing large language models for automated penetration testing. In: Proceedings of the 33rd USENIX Conference on Security Symposium. SEC ’24. USENIX Association, USA
  9. Garmany B, Stoffel M, Gawlik R, Koppe P, Blazytko T, Holz T (2018) Towards automated generation of exploitation primitives for web browsers. In: Proceedings of the 34th Annual Computer Security Applications Conference. ACSAC ’18, pp. 300–312. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3274694.3274723 · doi:10.1145/3274694.3274723
  10. Heelan S, Melham T, Kroening D (2019) Gollum: Modular and greybox exploit generation for heap overflows in interpreters. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. CCS ’19, pp. 1689–1706. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3319535.3354224 · DOI
  11. Hu H, Chua ZL, Adrian S, Saxena P, Liang Z (2015) Automatic generation of data-oriented exploits. In: Proceedings of the 24th USENIX Conference on Security Symposium. SEC’15, pp. 177–192. USENIX Association, USA
  12. Huang S-K, Huang M-H, Huang P-Y, Lai C-W, Lu H-L, Leong W-M (2012) Crax: Software crash analysis for automatic exploit generation by modeling attacks as symbolic continuations. In: Proceedings of the 2012 IEEE Sixth International Conference on Software Security and Reliability. SERE ’12, pp. 78–87. IEEE Computer Society, USA https://doi.org/10.1109/SERE.2012.20 · doi:10.1109/sere.2012.20
  13. Ispoglou KK, AlBassam B, Jaeger T, Payer M (2018) Block oriented programming: Automating data-only attacks. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. CCS ’18, pp. 1868–1882. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3243734.3243739 · DOI
  14. Kaniewski S, Schmidt F, Enzweiler M, Menth M, Heer T (2025) A systematic literature review on detecting software vulnerabilities with large language models. arXiv preprint arXiv:2507.22659https://doi.org/10.48550/arXiv.2507.22659 · arXiv · doi:10.48550/arxiv.2507.22659
  15. Khan S (2024) Ll-xss: End-to-end generative model-based xss payload creation. In: 2024 21st Learning and Technology Conference (L&T), pp. 121–126 https://doi.org/10.1109/LT60077.2024.10469151 · DOI
  16. Klischies D, Mackensen P, Moonsamy V (2025) Vulnerability where art thou? an investigation of vulnerability management in android smartphone chipsets. NDSS Symposium. https://doi.org/10.14722/ndss.2025.241161 · doi:10.14722/ndss.2025.241161
  17. Kurmus A, Mambretti A, Sorniotti A, Lenders V, Pfammatter D, Tellenbach B (2025) SoK: Automating kernel vulnerability discovery and exploit generation. 19th USENIX WOOT Conference on Offensive Technologies (WOOT ’25). pp 283–302
  18. Liu J, An H, Li J, Liang H (2022) Detecting exploit primitives automatically for heap vulnerabilities on binary programs. https://doi.org/10.48550/arXiv.2212.13990 · arXiv · doi:10.48550/arxiv.2212.13990
  19. Peng W, Ye L, Du X, Zhang H, Zhan D, Zhang Y, Guo Y, Zhang C (2025) PwnGPT: Automatic exploit generation based on large language models. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11481–11494. Association for Computational Linguistics, Vienna, Austria https://doi.org/10.18653/v1/2025.acl-long.562 · DOI
  20. Ray S, Pan R, Gu Z, Du K, Feng S, Ananthanarayanan G, Netravali R, Jiang J (2025) Metis: fast quality-aware rag systems with configuration adaptation. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. pp 606–622 · DOI
  21. Shao M, Xi H, Rani N, Udeshi M, Putrevu VSC, Milner K, Dolan-Gavitt B, Shukla SK, Krishnamurthy P, Khorrami F, Karri R, Shafique M (2025) CRAKEN: Cybersecurity LLM agent with knowledge-based execution. arXiv preprint arXiv:2505.17107https://doi.org/10.48550/arXiv.2505.17107 · DOI
  22. Turpin M, Michael J, Perez E, Bowman S (2023) Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Adv Neural Inf Process Syst 36:74952–74965 · doi:10.52202/075280-3275 · Paper page
  23. Wang Y, Zhang C, Xiang X, Zhao Z, Li W, Gong X, Liu B, Chen K, Zou W (2018) Revery: From proof-of-concept to exploitable. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. CCS ’18, pp. 1914–1927. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3243734.3243847 · doi:10.1145/3243734.3243847
  24. Wang D, Zhou G, Chen L, Li D, Miao Y (2024) Prophetfuzz: Fully automated prediction and fuzzing of high-risk option combinations with only documentation via large language model. In: Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. CCS ’24, pp. 735–749. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/3658644.3690231 · DOI
  25. Wu W, Chen Y, Xu J, Xing X, Gong X, Zou W (2018) Fuze: towards facilitating exploit generation for kernel use-after-free vulnerabilities. In: Proceedings of the 27th USENIX Conference on Security Symposium. SEC’18, pp. 781–797. USENIX Association, USA
  26. Wu Q, Bansal G, Zhang J, Wu Y, Li B, Zhu E, Jiang L, Zhang X, Zhang S, Liu J et al.: (2024) Autogen: Enabling next-gen llm applications via multi-agent conversations. In: First Conference on Language Modeling
  27. Xiao Z, Wang Q, Li Y, Chen S (2025) Prompt to pwn: Automated exploit generation for smart contracts. arXiv preprint arXiv:2508.01371https://doi.org/10.48550/arXiv.2508.01371 · arXiv · doi:10.48550/arxiv.2508.01371
  28. Xu J, Stokes JW, McDonald G, Bai X, Marshall D, Wang S, Swaminathan A, Li Z (2024) Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv preprint arXiv:2403.01038https://doi.org/10.48550/arXiv.2403.01038 · DOI
  29. Xu C, Guan S, Greene D, Kechadi M (2024) Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244 · arXiv
  30. Yang G, Zhou Y, Chen X, Zhang X, Han T, Chen T (2023) Exploitgen: Template-augmented exploit code generation based on codebert. J Syst Softw 197:111577. https://doi.org/10.1016/j.jss.2022.111577 · DOI
  31. Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K, Cao Y (2023) ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR)
  32. Zhu Y, Kellermann A, Gupta A, Li P, Fang R, Bindu R, Kang D (2024) Teams of LLM agents can exploit zero-day vulnerabilities. https://doi.org/10.48550/arXiv.2406.01637 · DOI

Metadata

Authors
Wei, C., Geng, Y., Wang, Y., Wu, Q., Huang, J., Wu, Q., & Wei, Q.
Venue
Cybersecurity
Published
2026-09-28
Volume
9
Issue
1
ISSN
2523-3246
Cited by
0
References
16
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation. https://app.cochat.ai/papers/10.1186/s42400-026-00649-5
The paper
Wei, C., Geng, Y., Wang, Y., Wu, Q., Huang, J., Wu, Q., & Wei, Q. (2026). Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation. Cybersecurity. https://doi.org/10.1186/s42400-026-00649-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.