AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Open accessAbstract
The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task. We evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking, (2) simple universal jailbreak templates can be adapted to effectively jailbreak agents, and (3) these jailbreaks enable coherent and malicious multi-step agent behavior and retain model capabilities. To enable simple and reliable evaluation of attacks and defenses for LLM-based agents, we publicly release AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AgentHarm assesses potentially harmful behaviors, while AgentDojo evaluates resilience against prompt injection attacks.
“Several benchmarks also support safety and robustness testing-for example, AgentHarm [5] assesses potentially harmful behaviors, while AgentDojo [17] evaluates resilience against prompt injection attacks.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · see citing works
Self-citationInjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2403.02691 · Cited by 143
ChemCrow: Augmenting large-language models with chemistry tools
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2023) · arXiv · doi:10.48550/arxiv.2304.05376 · Cited by 122 · see citing works
Emergent autonomous scientific research capabilities of large language models
Boiko, D. A., MacKnight, R., & Gomes, G. (2023) · arXiv · doi:10.48550/arxiv.2304.05332 · Cited by 72 · see citing works
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
Low-Resource Languages Jailbreak GPT-4Preprint
Yong, Z.-X., Menghini, C., & Bach, S. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02446 · Cited by 21 · see citing works
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2309.15817 · Cited by 19 · see citing works
Evaluating Language-Model Agents on Realistic Autonomous Tasks
Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Miles, L. H., Lin, T. R., Wijk, H., Burget, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11671 · Cited by 15 · see citing works
GAIA: a benchmark for General AI Assistants
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2311.12983 · Cited by 12 · see citing works
Calibrating LLM-Based Evaluator
Liu, Y., Yang, T., Huang, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., & Zhang, Q. (2023) · arXiv · doi:10.48550/arxiv.2309.13308 · Cited by 12 · see citing works
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 10 · see citing works
Evil Geniuses: Delving into the Safety of LLM-based Agents
Tian, Y., Yang, X., Zhang, J., Dong, Y., & Su, H. (2023) · arXiv · doi:10.48550/arxiv.2311.11855 · Cited by 10 · cited 1× · see citing works
“Tian et al. (2023) investigate harm in multi-agent LLM settings, but do not consider tool use or multi-turn interactions.”
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
Zhang, B., Tan, Y., Shen, Y., Salem, A., Backes, M., Zannettou, S., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.20859 · Cited by 5 · see citing works
LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks YetPreprint
Li, N., Han, Z., Steneker, I., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.15221 · Cited by 4 · see citing works
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language ModelsPreprint
Zhang, A. K., Perry, N., Dulepet, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.08926 · Cited by 3 · see citing works
Exploiting Novel GPT-4 APIsPreprint
Pelrine, K., Taufeeque, M., Zając, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.14302 · Cited by 3 · see citing works
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
Xiang, Z., Zheng, L., Li, Y., Hong, J., Li, Q., Xie, H., Zhang, J., Xiong, Z., Xie, C., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09187 · Cited by 2 · see citing works
Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.04682 · Cited by 2 · see citing works
BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM SafeguardsPreprint
Dorn, D., Variengien, A., Segerie, C.-R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.01364 · Cited by 2 · see citing works
The Dark Side of Function Calling: Pathways to Jailbreaking Large Language ModelsPreprint
Wu, Z., Gao, H., He, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.17915 · Cited by 2 · see citing works
PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-RailsPreprint
Mangaokar, N., Hooda, A., Choi, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.15911
Flagged first, then by citations
Research works printed but not linked: 12
- Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned llms with simple adaptive attacks. ICLR, 2025.
- Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. In NeurIPS, 2024.
- Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In NeurIPS Datasets and Benchmarks Track, 2024.
- Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. In NeurIPS Datasets and Benchmarks, 2024.
- Tom Gibbs, Ethan Kosak-Hine, George Ingebretsen, Jason Zhang, Julius Broomfield, Sara Pieri, Reihaneh Iranmanesh, Reihaneh Rabbany, and Kellin Pelrine. Emerging vulnerabilities in fron- tier models: Multi-turn jailbreak attacks, 2024. URL https://arxiv.org/abs/2409. 00137.
- Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pages 79-90, 2023.
- Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standard- ized evaluation framework for automated red teaming and robust refusal. In ICML, 2024.
- Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Han- lin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, pages 26837-26867. PMLR, 2023.
- Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dHng2O0Jjr.
- Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongreject for empty jail- breaks. In NeurIPS, 2024.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
- Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In NeurIPS, 2024.
Cited by
Sorted by citations
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
“For example, in the assessment of web agents, researchers built web simulators (MiniWoB [85], WebShop [103], WebArena [126], etc.) where the behavior of an agent (clicking links, filling forms) can be programmed to verify the correct sequence.”
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3 · see citing works
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
Wang, Z., Schiller, N., Li, H., Narayana, S. S., Nasr, M., Carlini, N., Qi, X., Wallace, E., Bursztein, E., Invernizzi, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.11086 · Cited by 3
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M. (2025) · arXiv · doi:10.48550/arxiv.2502.08586 · Cited by 2 · see citing works
“Two recent papers release platforms for testing LLM agent security (Debenedetti et al., 2024; Andriushchenko et al., 2024b).”
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., & Chaudhuri, K. (2025) · arXiv · doi:10.48550/arxiv.2504.18575 · Cited by 1 · see citing works
“Andriushchenko et al. (2024) provides a benchmark of harmful behaviors for tool-use agents, together with an evaluation method, and shows that many existing LLMs are susceptible to these requests.”
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04510
“There are further risks in agentic [34, 104, 96, 72, 18, 31, 70, 115, 4, 10, 12, 14, 38, 42, 63, 81, 93, 101, 107, 114, 84, 2] versions of H ETEROGENEOUS S WARMS where one model could be biased or malicious.”
Multi-Agent Systems Execute Arbitrary Malicious Code
Triedman, H., Jha, R., & Shmatikov, V. (2025) · arXiv · doi:10.48550/arxiv.2503.12188
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Vijayvargiya, S., Soni, A. B., Zhou, X., Wang, Z. Z., Dziri, N., Neubig, G., & Sap, M. (2025) · arXiv · doi:10.48550/arxiv.2507.06134
“…[42] ✓ ✗ PrivacyLens [28] ✓ ✗ Dissecting Adversarial [34] ✓ ✗ Infrastructure for AI [4] ✗ ✓ R-Judge [39] ✗ ✓ Trembling House [19] ✗ & ✗ AgentHarm [2] ✗ & ✗ WildTeaming [11] ✗ & ✗ SafetyPrompts [25] ✗ & ✓ ST-WebAgentBench [15] ✓ ✓ Frontier Models [18] ✓ ✓ SafeArena [31] ✓ & ✗ Haicosystem [46] ✗ &…”
LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
Zhao, L., Bhaskar, A., & Dobriban, E. (2026) · arXiv · doi:10.48550/arxiv.2605.17986
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Dev, S., Sloan, A., Kavner, J., Kong, N., & Sandler, M. (2026) · arXiv · doi:10.48550/arxiv.2603.05399
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594
- Authors
- Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., Winsor, E., Wynne, J., Gal, Y., & Davies, X.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.09024
- OpenAlex
- W7128378279
- Cited by
- 11
- References
- 24
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Funders
- National Science Foundation
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (214838)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Access Control and Trust
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- telos, LLM agents, biosecurity, chemical security, cybersecurity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
