CoChat
Sign Up
PreprintCited by 1Multi-Agent Systems and NegotiationNatural Language Processing TechniquesDistributed Control Multi-Agent Systems

On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents

Open access
Huang, Jen-tse, Zhou, Jiaxu, Jin, Tailin, Zhou, Xuhui, Chen, Zixi, Wang, Wenxuan, Yuan, Youliang, Lyu, Michael R. +1 more·2024·arXiv
DOI 10.48550/arxiv.2408.00989arXiv 2408.00989
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents, each focusing on a specific domain. However, the impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored. This paper investigates: (1) What is the resilience of various system structures (e.g., A$\rightarrow$B$\rightarrow$C, A$\leftrightarrow$B$\leftrightarrow$C) under faulty agents, on different downstream tasks? (2) How can we increase system resilience to defend against these agents? To simulate faulty agents, we propose two approaches--AutoTransform and AutoInject--which introduce mistakes into the agents' responses. Experiments on four downstream tasks using six systems show that the "hierarchical" structure, i.e., A$\rightarrow$(B$\leftrightarrow$C), exhibits superior resilience with the lowest performance drop of 5.5%, compared to 10.5% and 23.7% of other two structures. To further improve resilience, we introduce (1) Challenger, that introduces a mechanism for each agent to challenge others' outputs, and (2) Inspector, an additional agent to review and correct messages, recovering up to 96.4% errors made by faulty agents. Our code and data are available at https://github.com/CUHK-ARISE/MAS-Resilience.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.00989

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Attacks on AI agents include attacks on overseer agents, attempts to thwart cooperation, and leakage of information that could enable collusion.

    “This includes attacks that have direct relevance for ensuring safety, such as attacks on overseer agents (see Case Study 13), attempts to thwart cooperation (Huang et al., 2024; Lamport et al., 1982), and the leakage of information (accidentally or deliberately) that could be used to enable collusion (Motwani et al., 2024).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Huang et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 1

Research works printed but not linked: 61

  1. Alexy, O. How flat can it get? from better at flatter to the promise of the decentralized, boundaryless organization. Journal of Organization Design, 11(1):31-36, 2022.
  2. Alliger, G. M., Cerasoli, C. P., Tannenbaum, S. I., and Vessey, W. B. Team resilience: How teams flourish un- der pressure. Organizational Dynamics, 44(3):176-184, 2015.
  3. Amayuelas, A., Yang, X., Antoniades, A., Hua, W., Pan, L., and Wang, W. Multiagent collaboration attack: In- vestigating adversarial attacks in large language model collaborations via debate. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
  4. Becker, J. Multi-agent large language models for conver- sational task-solving. arXiv preprint arXiv:2410.22932, 2024. · arXiv
  5. Boin, A. and Van Eeten, M. J. The resilient organization. Public Management Review, 15(3):429-445, 2013.
  6. Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024.
  7. Chen, A., Wu, Y., Zhang, J., Yang, S., Huang, J.-t., Wang, K., Wang, W., and Wang, S. A survey on the safety and security threats of computer-using agents: Jarvis or ultron? arXiv preprint arXiv:2505.10924, 2025a. · arXiv
  8. Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., and Shi, Y. Autoagents: A framework for automatic agent generation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024a.
  9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  10. Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., et al. Agentverse: Facilitating multi-agent collaboration and exploring emer- gent behaviors. In The Twelfth International Conference on Learning Representations, 2024b.
  11. Chen, W., You, Z., Li, R., Guan, Y., Qian, C., Zhao, C., Yang, C., Xie, R., Liu, Z., and Sun, M. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. In The Thirteenth International Conference on Learning Representations, 2025b.
  12. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the Forty-first International Conference on Machine Learn- ing, 2024.
  13. Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q., Wang, Y., Jiang, J., and Lin, M. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. In Forty-first International Conference on Machine Learning, 2024.
  14. Guha, N., Nyarko, J., Ho, D., Ré, C., Chilton, A., Chohlas- Wood, A., Peters, A., Waldon, B., Rockmore, D., Zam- brano, D., et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large lan- guage models. Advances in Neural Information Process- ing Systems, 36, 2023.
  15. Hartwig, A., Clarke, S., Johnson, S., and Willis, S. Work- place team resilience: A systematic review and concep- tual development. Organizational Psychology Review, 10 (3-4):169-200, 2020.
  16. He, J., Wang, T., Xiong, D., and Liu, Q. The box is in the pen: Evaluating commonsense reasoning in neural ma- chine translation. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pp. 3662-3672, 2020.
  17. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multi- task language understanding. In The Ninth International Conference on Learning Representations, 2021.
  18. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Meta programming for a multi-agent collabora- tive framework. In The Twelfth International Conference on Learning Representations, 2024.
  19. Jiao, W., Wang, W., Huang, J.-t., Wang, X., and Tu, Z. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 2023. · arXiv
  20. Ju, T., Wang, Y., Ma, X., Cheng, P., Zhao, H., Wang, Y., Liu, L., Xie, J., Zhang, Z., and Liu, G. Flooding spread of manipulated knowledge in llm-based multi-agent com- munities. arXiv preprint arXiv:2407.07791, 2024. · arXiv
  21. Lee, C., Xia, C. S., Huang, J.-t., Zhu, Z., Zhang, L., and Lyu, M. R. A unified debugging approach via llm-based multi-agent synergy. arXiv preprint arXiv:2404.17153, 2024. · arXiv
  22. Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2023.
  23. Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., Ma, W., and Liu, Y. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957, 2024. · arXiv
  24. Liang, T., He, Z., Huang, J.-t., Wang, W., Jiao, W., Wang, R., Yang, Y., Tu, Z., Shi, S., and Wang, X. Leverag- ing word guessing games to assess the intelligence of large language models. arXiv preprint arXiv:2310.20499, 2023. · arXiv
  25. Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, 2024.
  26. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3214- 3252, 2022.
  27. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2023.
  28. Liu, Z., Anand, A., Zhou, P., Huang, J.-t., and Zhao, J. Interintent: Investigating social intelligence of llms via intention understanding in an interactive game context. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
  29. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024.
  30. Mao, J., Meng, F., Duan, Y., Yu, M., Jia, X., Fang, J., Liang, Y., Wang, K., and Wen, Q. Agentsafe: Safeguarding large language model-based multi-agent systems via hierarchi- cal data management. arXiv preprint arXiv:2503.04392, 2025. · arXiv
  31. Mihm, J., Loch, C. H., Wilkinson, D., and Huberman, B. A. Hierarchical structure and search in complex organiza- tions. Management science, 56(5):831-848, 2010.
  32. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, 2022.
  33. Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medi- cal domain question answering. In Conference on health, inference, and learning, pp. 248-260. PMLR, 2022.
  34. Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
  35. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-3448, 2022.
  36. Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. Chatdev: Commu- nicative agents for software development. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174-15186, 2024.
  37. Qian, C., Xie, Z., Wang, Y., Liu, W., Dang, Y., Du, Z., Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large-language- model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations, 2025a.
  38. Qian, C., Xie, Z., Wang, Y., Liu, W., Dang, Y., Du, Z., Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large-language- model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations, 2025b.
  39. Sellam, T., Das, D., and Parikh, A. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, 2020.
  40. Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., and Kan, M.-Y. Reliability testing for natural language processing systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4153- 4169, 2021.
  41. Tian, Y., Yang, X., Zhang, J., Dong, Y., and Su, H. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855, 2023. · arXiv
  42. Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., and Nguyen, H. D. Multi-agent collab- oration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322, 2025. · arXiv
  43. Wang, K., Zhang, G., Zhou, Z., Wu, J., Yu, M., Zhao, S., Yin, C., Fu, J., Yan, Y., Luo, H., et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 2025a. · arXiv
  44. Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440-9450, 2024a.
  45. Wang, S., Zhang, G., Yu, M., Wan, G., Meng, F., Guo, C., Wang, K., and Wang, Y. G-safeguard: A topology- guided security lens and treatment on llm-based multi- agent systems. arXiv preprint arXiv:2502.11127, 2025b. · arXiv
  46. Wang, X., Xiao, Y., Huang, J.-t., Yuan, S., Xu, R., Guo, H., Tu, Q., Fei, Y., Leng, Z., Wang, W., Chen, J., Li, C., and Yanghua, X. Incharacter: Evaluating personality fidelity in role-playing agents through psychological in- terviews. In The 62nd Annual Meeting of the Association for Computational Linguistics, 2024b.
  47. Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self- collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 257-279, 2024c.
  48. Wu, M., Yuan, Y., Haffari, G., and Wang, L. (perhaps) beyond human translation: Harnessing multi-agent col- laboration for translating ultra-long literary texts. arXiv preprint arXiv:2405.11804, 2024a. · arXiv
  49. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Li, B., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent con- versations. In First Conference on Language Modeling, 2024b.
  50. Yang, H. and Zhang, L. Communication and the optimal- ity of hierarchy in organizations. The Journal of Law, Economics, and Organization, 35(1):154-191, 2019.
  51. Yu, M., Fang, J., Zhou, Y., Fan, X., Wang, K., Pan, S., and Wen, Q. Llm-virus: Evolutionary jailbreak attack on large language models. arXiv preprint arXiv:2501.00055, 2024a. · arXiv
  52. Yu, M., Wang, S., Zhang, G., Mao, J., Yin, C., Liu, Q., Wen, Q., Wang, K., and Wang, Y. Netsafe: Exploring the topological safety of multi-agent networks. arXiv preprint arXiv:2410.15686, 2024b. · arXiv
  53. Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pang, L., Chen, T., Wang, K., Li, X., Zhang, Y., et al. A survey on trustworthy llm agents: Threats and countermeasures. arXiv preprint arXiv:2503.09648, 2025. · arXiv
  54. Yu, W., Hu, K., Pang, T., Du, C., Lin, M., and Fredrikson, M. Infecting llm agents via generalizable adversarial attack. In NeurIPS 2024 Workshop Red Teaming GenAI: What Can We Learn from Adversaries?, 2024c.
  55. Zhang, Z., Zhang, Y., Li, L., Gao, H., Wang, L., Lu, H., Zhao, F., Qiao, Y., and Shao, J. Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety. In The 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
  56. Zhou, W., Jiang, Y. E., Li, L., Wu, J., Wang, T., Qiu, S., Zhang, J., Chen, J., Wu, R., Wang, S., et al. Agents: An open-source framework for autonomous language agents. arXiv preprint arXiv:2309.07870, 2023. · arXiv
  57. Zhou, X., Su, Z., Eisape, T., Kim, H., and Sap, M. Is this the real life? is this just fantasy? the misleading success of simulating social interactions with llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024a.
  58. Zhou, X., Zhu, H., Mathur, L., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., et al. Sotopia: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024b.
  59. Zhou, Z., Li, Z., Zhang, J., Zhang, Y., Wang, K., Liu, Y., and Guo, Q. Corba: Contagious recursive blocking attacks on multi-agent systems based on large language models. arXiv preprint arXiv:2502.14529, 2025. · arXiv
  60. Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Gptswarm: Language agents as optimizable graphs. In The Forty-first International Conference on Machine Learning, 2024.
  61. Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Uni- versal and transferable adversarial attacks on aligned lan- guage models. arXiv preprint arXiv:2307.15043, 2023. · arXiv

Authors
Huang, J.-T., Zhou, J., Jin, T., Zhou, X., Chen, Z., Wang, W., Yuan, Y., Lyu, M. R., & Sap, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.00989
arXiv
2408.00989
OpenAlex
W4403808675
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Defense Advanced Research Projects Agency
  • Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Control Multi-Agent Systems
Keywords
multi-agent systems, large language models, malicious agents, agent cooperation, fault injection, error recovery

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review