The DOI matches its Crossref record; 5 of the 6 references it cites were found in a registry or an index.
AutoPenBench: A Vulnerability Testing Benchmark for Generative Agents
Open accessAbstract
Luca Gioacchini, Alexander Delsanto, Idilio Drago, Marco Mellia, Giuseppe Siracusano, Roberto Bifulco. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2025.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deployable autonomy can be measured by counting missing evidence, budget overruns, gate violations, unsafe execution, and incomplete audit trails.
“This turns deployable autonomy into a measurable claim: missing evidence, budget overruns, gate violations, unsafe execution, and incomplete audit trails can all be counted [162], [163].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [162]
Section 2.3 (page not specified) measures task progress through command and stage milestones and requires a proof-of-completion flag; Section 4.2 reports a task failure after the agent exceeded a step limit. These support counting some missing task evidence and step-limit overruns, but our copy does not establish measures for gate violations, unsafe execution, or incomplete audit trails. (LLM, conf=0.84)
Experimental — accuracy not yet measuredRobustness, calibration, and efficiency are central in high-stakes deployment.
“This follows established evaluation practice, where robustness, calibration, and efficiency are central in high-stakes deployment [107], [162], [163].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [162]
Section 5.2, Agent Consistency (page unavailable), says variability in agent performance affects reliability in real-world applications, supporting the robustness aspect in vulnerability testing. Section 4.2, Assisted Agent (page unavailable), mentions tackling subtasks more efficiently, but the paper does not establish calibration or the broader claim that all three are central to high-stakes deployment. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Wenliang Du. 2011. Seed: hands-on lab exercises for computer security education. IEEE Security & Pri- vacy, 9(5):70-73.
- OpenAI. 2024. Introducing OpenAI O1. OpenAI. 2025. Gpt-4. · DOI
- Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. Cognitive Architec- tures for Language Agents. · DOI
- Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Pee- tathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, and 8 others. 2024. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models. · DOI · Paper page
Cited by
Sorted by citations
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis
Chen, M., Cheng, H., Su, T., Chen, M., Cai, W., & Zou, H. (2026) · Electronics · doi:10.3390/electronics15102222
Kuzmin, A., Латохин, Д. В., Anistratenko, A. A., Афонин, Е. С., Лискин, А. В., & Иванов, А. М. (2026) · Sovremennye informacionnye tehnologii i IT-obrazovanie. · doi:10.25559/sitito.022.202601.13-27
GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks
Hadi, H. J., Khan, M. K., Ahmad, N., Yasmin, R., & Shoker, A. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3716998 · Cited by 1
Yinhang, Z., & Weihua, X. (2026) · 數據科學與人工智能學報 · doi:10.71411/dsai.2026.v1i1.1768
MAM-PT: A Memory-Augmented Middleware for Enhancing Automated Penetration Testing
Sun, W., Wang, W., Wu, W., & Wang, Y. (2026) · International Conference on Computer Science and Blockchain (CCSB) · doi:10.1109/ccsb71176.2026.11716413
Metadata
- Authors
- Gioacchini, L., Delsanto, A., Drago, I., Mellia, M., Siracusano, G., & Bifulco, R.
- Affiliations
- Politecnico di Torino (IT)
- NEC (Germany) (DE)
- Venue
- Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Published
- 2025
- Pages
- 1615-1624
- OpenAlex
- W4416037318
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 4
- FWCI
- 3.81
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
- Funders
- European Commission (PE00000014, 2022EP2L7H, PRIN-2022)
- NextGenerationEU
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- generative AI agents, vulnerability testing, benchmark, large language models, AI safety, adversarial testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
