Evil Geniuses: Delving into the Safety of LLM-based Agents
Open accessAbstract
Rapid advancements in large language models (LLMs) have revitalized in LLM-based agents, exhibiting impressive human-like behaviors and cooperative capabilities in various scenarios. However, these agents also bring some exclusive risks, stemming from the complexity of interaction environments and the usability of tools. This paper delves into the safety of LLM-based agents from three perspectives: agent quantity, role definition, and attack level. Specifically, we initially propose to employ a template-based attack strategy on LLM-based agents to find the influence of agent quantity. In addition, to address interaction environment and role specificity issues, we introduce Evil Geniuses (EG), an effective attack method that autonomously generates prompts related to the original role to examine the impact across various role definitions and attack levels. EG leverages Red-Blue exercises, significantly improving the generated prompt aggressiveness and similarity to original roles. Our evaluations on CAMEL, Metagpt and ChatDev based on GPT-3.5 and GPT-4, demonstrate high success rates. Extensive evaluation and discussion reveal that these agents are less robust, prone to more harmful behaviors, and capable of generating stealthier content than LLMs, highlighting significant safety challenges and guiding future research. Our code is available at https://github.com/T1aNS1R/Evil-Geniuses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
Cited by
Sorted by citations
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 209 · see citing works
LLM-based agentic systems in medicine and healthcare
Qiu, J., Lam, K., Li, G., Acharya, A., Wong, T. Y., Darzi, A., Yuan, W., & Topol, E. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00944-1 · Cited by 168 · see citing works
When Embodied AI Meets Industry 5.0: Human-Centered Smart Manufacturing
Xu, J., Sun, Q.-Y., Han, Q.-L., & Tang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125327 · Cited by 102 · see citing works
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147 · see citing works
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey
Masterman, T., Besen, S., Sawtell, M., & Chao, A. (2024) · arXiv · doi:10.48550/arxiv.2404.11584 · Cited by 35 · see citing works
“Moreover, agents have specifically been shown to be “less robust, prone to more harmful behaviors, and capable of generating stealthier content than LLMs, highlighting significant safety challenges” [25].”
BackgroundPrompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
Lee, D., & Tiwari, M. (2024) · arXiv · doi:10.48550/arxiv.2410.07283 · Cited by 18 · see citing works
Pingua, B., Murmu, D., Kandpal, M., Rautaray, J., Mishra, P., Barik, R. K., & Saikia, M. J. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2374 · Cited by 24 · see citing works
AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models
Shu, D., Zhang, C., Jin, M., Zhou, Z., & Li, L. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3748239.3748242 · Cited by 33 · see citing works
“These prompts might involve complex scenarios, role-playing, or specific phrasing that confuses the model’s understanding of what is permissible [1].”
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
Self-citationMulti-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
Large Language Models in Cyberattacks
Лебедь, С. В., Namiot, D., Zubareva, E. V., Хенкин, П. В., Воробьева, А. А., & Свичкарь, Д. А. (2024) · Doklady Mathematics · doi:10.1134/s1064562425700012 · Cited by 6 · see citing works
“We can draw a unambiguous conclusion that development of LLM technologies and reduced requirements to computing resources will lead to larger number of such services and models, which creates substantial risks for cyber-security and makes the access for intruders easier [13].”
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.09024 · Cited by 11 · see citing works
“Tian et al. (2023) investigate harm in multi-agent LLM settings, but do not consider tool use or multi-turn interactions.”
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4 · see citing works
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.10019 · Cited by 2 · see citing works
“Equipped with tool usage and interacting with environments, LLM agents unlock novel real-world safety issues, opening up research on behavioral safety (Ruan et al., 2024; Naihin et al., 2023; Tian et al., 2023; Zhan et al., 2024) of LLM agents within diverse environments.”
BackgroundCommercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M. (2025) · arXiv · doi:10.48550/arxiv.2502.08586 · Cited by 2 · see citing works
A Review of the Application of Prompt Engineering in the Safety of Large Language Models
Guo, H., Zhang, L., Feng, X., & Zheng, Q. (2024) · International Conference on Information Education and Artificial Intelligence (ICIEAI) · doi:10.1145/3724504.3724574 · Cited by 3 · see citing works
“Here, the attack model learns from feedback to adjust its policies [50].”
Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Models
Ke, S.-W., Lai, G.-Y., Fang, G.-L., & Kao, H.-Y. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0887-7_33 · Cited by 1 · see citing works
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
A closer look at how large language models ‘trust' humans: patterns and biases
Lerman, V., & Dover, Y. (2026) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2025.1113
“%) having adopted some sort of AI tools during 2024.”
ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Mashkif, N., & Shlomov, S. (2024) · arXiv · doi:10.48550/arxiv.2410.06703 · Cited by 1 · see citing works
- Authors
- Tian, Y., Yang, X., Zhang, J., Dong, Y., & Su, H.
- Affiliations
- Tsinghua University (CN)
- Kuaishou (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.11855
- OpenAlex
- W4388926678
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- LLM agents, agent safety, prompt injection, agent robustness, camels
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
