The DOI matches its Crossref record; 313 of the 313 references it cites were found in a registry or an index.
A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges
Open accessAbstract
The pursuit of more intelligent and credible autonomous systems, akin to human society, has been a long-standing endeavor for humans. Leveraging the exceptional reasoning and planning capabilities of large language models (LLMs), LLM-based agents have been proposed and have achieved remarkable success across a wide array of tasks. Notably, LLM-based multi-agent systems (MAS) are considered a promising pathway towards realizing general artificial intelligence that is equivalent to or surpasses human-level intelligence. In this paper, we present a comprehensive survey of these studies, offering a systematic review of LLM-based MAS. Adhering to the workflow of LLM-based multi-agent systems, we synthesize a general structure encompassing five key components: profile, perception, self-action, mutual interaction, and evolution. This unified framework encapsulates much of the previous work in the field. Furthermore, we illuminate the extensive applications of LLM-based MAS in two principal areas: problem-solving and world simulation. Finally, we discuss in detail several contemporary challenges and provide insights into potential future directions in this domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MAS, in which multiple autonomous agents collaborate as a team, are emerging as a powerful paradigm for biomedical and clinical data analysis [1,2].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, current RAG models face challenges related to frequent knowledge updates [2,11,21,22].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluating multi-agent collaboration in LLM-based systems requires different methodologies compared to traditional reinforcement learning-driven coordination [7, 48, 89].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“Feedback information can take on multiple forms, such as prompt contexts [240], embeddings [287], tokens [288, 291–293].”
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680 · cited 1×
“This information is then converted into intermediate representations through perception modules, which then determine the agent’s autonomous decision-making outcomes and behavioral responses [62, 63].”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Self-citationOvercoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“(2) Hallucination : LLMs can produce information that sounds plausible but is factually incorrect or nonsensical, known as hallucination [199].”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 6×
“…external knowledge bases to enhance information accuracy [143, 238], increasing model transparency to foster understanding of decision processes [86, 202, 203], developing fact-checking systems to verify outputs [200, 201], and designing hallucination detection tasks [325, 326] to evaluate and…”
InfluentialTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 3×
“Instruct-GPT [205] learns by collecting evaluations from human annotators on the model’s output, which include preference rankings for the text generated by the model, serving as a feedback signal.”
Research works printed but not linked: 86
- A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian et al., Do as i can, not as i say: Grounding language in robotic affordances. in Conference on robot learning (PMLR, Atlanta, GA, USA, 2023), pp. 287–318
- K. Li, A.K. Hopkins, D. Bau, F. Viégas, H. Pfister, M. Wattenberg, Emergent world representations: Exploring a sequence model trained on a synthetic task. in The Eleventh International Conference on Learning Representations (Kigali, Rwanda, 2023)
- N. Renella, M. Eger, Towards automated video game commentary using generative ai. in Proceedings of the Experimental Artificial Intelligence in Games Workshop co-located with the 19th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2023) (AAAI Press, Washington, DC, USA, 2023)
- Z. Ma, Y. Mei, Z. Su, Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. in AMIA Annual Symposium Proceedings (American Medical Informatics Association, New Orleans, LA, 2023). pp. 1105
- G. Weiss, Multiagent systems: a modern approach to distributed artificial intelligence (MIT Press, Cambridge, MA, United States, 1999), pp. 547
- H.V.D. Parunak, Applications of distributed artificial intelligence in industry. Found. Distrib. Artif. Intell. 2(1), 18 (1996)
- I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, R. Fergus, Collaborating with language models for embodied reasoning. in NeurIPS 2022 Foundation Models for Decision Making Workshop (New Orleans, Louisiana, United States of America, 2022)
- S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S.K.S. Yau, Z. Lin, L. Zhou et al., Metagpt: Meta programming for multi-agent collaborative framework. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J.B. Tenenbaum, T. Shu, C. Gan, Building cooperative embodied agents modularly with large language models. in NeurIPS 2023 Foundation Models for Decision Making Workshop (New Orleans, Louisiana, United States, 2023)
- Y. Du, S. Li, A. Torralba, J.B. Tenenbaum, I. Mordatch, Improving factuality and reasoning in language models through multiagent debate. in Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research (Vienna, Austria, 2024). 235, pp. 11733–11763. PMLR.
- C.M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, Z. Liu, Chateval: Towards better llm-based evaluators through multi-agent debate. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- J. Light, M. Cai, S. Shen, Z. Hu, Avalonbench: Evaluating llms playing the game of avalon. in NeurIPS 2023 Foundation Models for Decision Making Workshop (2023) (New Orleans, United States, 2023)
- G. Mukobi, H. Erlebach, N. Lauffer, L. Hammond, A. Chan, J. Clifton, Welfare diplomacy: Benchmarking language model cooperation. in NeurIPS 2023 Socially Responsible Language Modelling Research workshop (New Orleans, United States, 2023)
- G.V. Aher, R.I. Arriaga, A.T. Kalai, Using large language models to simulate multiple humans and replicate human subject studies. in International Conference on Machine Learning (PMLR, Honolulu, Hawaii, USA, 2023), pp. 337–371
- A. Zhang, L. Sheng, Y. Chen, H. Li, Y. Deng, X. Wang, T.S. Chua, On generative agents in recommendation. in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (Association for Computing Machinery, New York, NY, USA, 2024). SIGIR ’24, pp. 1807–1817. https://doi.org/10.1145/3626772.3657844 · DOI
- L. Wang, J. Zhang, X. Chen, Y. Lin, R. Song, W.X. Zhao, J.R. Wen, Recagent: A novel simulation paradigm for recommender systems. arXiv preprint (2023) arXiv:2306.0255
- M. Shridhar, X. Yuan, M.A. Côté, Y. Bisk, A. Trischler, M. Hausknecht, Alfworld: Aligning text and embodied environments for interactive learning. International Conference on Learning Representation (2021)
- D. Driess, F. Xia, M.S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., Palm-e: An embodied multimodal language model. in Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA, 2023). ICML’23. JMLR.org 340, 20.
- W. Tan, Z. Ding, W. Zhang, B. Li, B. Zhou, J. Yue, H. Xia, J. Jiang, L. Zheng, X. Xu et al., Towards general computer control: A multimodal agent for red dead redemption ii as a case study. in ICLR 2024 Workshop on Large Language Model (LLM) Agents (Vienna, Austria, 2024).
- S. Reed, K. Zolna, E. Parisotto, S.G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J.T. Springenberg et al., A generalist agent. Transactions on Machine Learning Research (2022).
- S. Zheng, Y. Feng, Z. Lu et al., Steve-eye: Equipping llm-based embodied agents with visual perception in open worlds. in The Twelfth International Conference on Learning Representations (Kigali, Rwanda, 2023)
- P.F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, Deep reinforcement learning from human preferences. Adv. Neural Inf. Process. Syst. 30 (2017). pp. 4302-4310
- J. Wei, M. Bosma, V.Y. Zhao, K. Guu, A.W. Yu, B. Lester, N. Du, A.M. Dai, Q.V. Le, Finetuned language models are zero-shot learners. in International Conference on Learning Representations (2022) (Virtual Event).
- W. Li, L. Zhu, L. Wen, Y. Yang, Decap: Decoding clip latents for zero-shot captioning via text-only training. in International Conference on Learning Representations (2023) (Kigali, Rwanda, 2023).
- S. Zhao, X. Wang, L. Zhu, Y. Yang, Test-time adaptation with clip reward for zero-shot generalization in vision-language models. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- J. Chen, H. Guo, K. Yi, B. Li, M. Elhoseiny, Visualgpt: Data-efficient image captioning by balancing visual input and linguistic knowledge from pretraining. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, New Orleans, LA, USA, 2022), pp. 18030–18040
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., An image is worth 16x16 words: Transformers for image recognition at scale. in International Conference on Learning Representations (2021) (Vienna, Austria).
- A. Van Den Oord, O. Vinyals et al., Neural discrete representation learning. Adv. Neural Inf. Process. Syst. 30 (2017). pp. 6309-6318
- S. Mehta, M. Rastegari, Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. in International Conference on Learning Representations (2022) (Virtual Event).
- J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. in International conference on machine learning (PMLR, Honolulu, Hawaii, USA, 2023), pp. 19730–19742
- D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024).
- Y. Su, T. Lan, H. Li, J. Xu, Y. Wang, D. Cai, Pandagpt: One model to instruction-follow them all. in Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! (Association for Computational Linguistics, Prague, Czech Republic, 2023), pp. 11-23.
- Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, F. Wei, Kosmos-2: Grounding multimodal large language models to the world. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024).
- L. Qian, J. Li, Y. Wu, Y. Ye, H. Fei, T.S. Chua, Y. Zhuang, S. Tang, Momentor: Advancing video large language model with fine-grained temporal reasoning. in Forty-first International Conference on Machine Learning (Vienna, Austria, 2024).
- Z. Yang, G. Chen, X. Li, W. Wang, Y. Yang, Doraemongpt: Toward understanding dynamic scenes with large language models. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024).
- L.H. Marshall, H.W. Magoun, Discoveries in the human brain: neuroscience prehistory, brain structure, and function (Springer Science & Business Media, Humana Totowa, NJ, 2013)
- L. Zheng, R. Wang, X. Wang, B. An, Synapse: Trajectory-as-exemplar prompting with memory for computer control. in The Twelfth International Conference on Learning Representations (Messe Wien Exhibition and Congress Center, Vienna, Austria, 2023)
- J. Kang, R. Laroche, X. Yuan, A. Trischler, X. Liu, J. Fu, Think before you act: Decision transformers with internal working memory. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024).
- A. Modarressi, A. Imani, M. Fayyaz, H. Schütze, Ret-llm: Towards a general read-write memory for large language models. in ICLR 2024 Workshop: How Far Are We From AGI (Vienna, Austria, 2024)
- T. Silver, V. Hariprasad, R.S. Shuttleworth, N. Kumar, T. Lozano-Pérez, L.P. Kaelbling, Pddl planning with pretrained large language models. in NeurIPS 2022 foundation models for decision making workshop (New Orleans, Louisiana, USA, 2022)
- Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, H. Soh, Translating natural language to planning goals with large-language models. in The International Journal of Robotics Research (2020). 2019, pp. 1
- M. Tsimpoukelli, J.L. Menick, S. Cabi, S. Eslami, O. Vinyals, F. Hill, Multimodal few-shot learning with frozen language models. Adv. Neural Inf. Process. Syst. 34, 200–212 (2021)
- H. Liu, C. Li, Q. Wu, Y.J. Lee, Visual instruction tuning. Adv. Neural Inf. Process. Syst. 36(2024). pp. 34892-34916
- L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, G. Neubig, Pal: Program-aided language models. in International Conference on Machine Learning (PMLR, Honolulu, Hawaii, USA, 2023), pp. 10764–10799
- X. Cheng, D. Luo, X. Chen, L. Liu, D. Zhao, R. Yan, Lift yourself up: Retrieval-augmented text generation with self-memory. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 43780-43799
- W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, M. Jiang, Generate rather than retrieve: Large language models are strong context generators. in The Eleventh International Conference on Learning Representations (Kigali, Rwanda, 2023)
- A. Asai, Z. Wu, Y. Wang, A. Sil, H. Hajishirzi, Self-rag: Learning to retrieve, generate, and critique through self-reflection. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- F. Xu, W. Shi, E. Choi, Recomp: Improving retrieval-augmented lms with compression and selective augmentation. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- E. Mitchell, C. Lin, A. Bosselut, C.D. Manning, C. Finn, Memory-based model editing at scale. in International Conference on Machine Learning (PMLR, Baltimore, Maryland, USA, 2022), pp. 15817–15831
- J. Maynez, S. Narayan, B. Bohnet, R. McDonald, On faithfulness and factuality in abstractive summarization. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, Online, 2022), pp. 1906-1919
- J. Thorne, A. Vlachos, Automated fact checking: Task formulations, methods and future directions. in Proceedings of the 27th International Conference on Computational Linguistics (Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018), pp. 3346-3359
- C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu et al., Lima: Less is more for alignment. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 55006-55021
- Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N.A. Smith, M. Ostendorf, H. Hajishirzi, Fine-grained human feedback gives better rewards for language model training. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 59008-59033
- Y.S. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, P. He, Dola: Decoding by contrasting layers improves factuality in large language models. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- P.C. Wason, P.N. Johnson-Laird, Psychology of reasoning: Structure and content. Harvard University Press, Cambridge, MA, USA, 1972. 86, pp. 246-252
- D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le et al., Least-to-most prompting enables complex reasoning in large language models. in The Eleventh International Conference on Learning Representations (Kigali, Rwanda, 2023)
- M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., Do as i can, not as i say: Grounding language in robotic affordances. in Conference on Robot Learning (Atlanta, GA, United States, 2023), pp. 287–318. PMLR
- S.S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, S. Tellex, Planning with large language models via corrective re-prompting. in NeurIPS 2022 Foundation Models for Decision Making Workshop (New Orleans, Louisiana, USA, 2022)
- T. Kojima, S.S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners. Adv. Neural Inf. Process. Syst. 35, 22199–22213 (2022)
- B.Y. Lin, Y. Fu, K. Yang, F. Brahman, S. Huang, C. Bhagavatula, P. Ammanabrolu, Y. Choi, X. Ren, Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 23813-23825
- S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, K. Narasimhan, Tree of thoughts: Deliberate problem solving with large language models. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 11809-11822
- W. Huang, P. Abbeel, D. Pathak, I. Mordatch, Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. in International Conference on Machine Learning (PMLR, Baltimore, Maryland, USA, 2022), pp. 9118–9147
- K. Nottingham, P. Ammanabrolu, A. Suhr, Y. Choi, H. Hajishirzi, S. Singh, R. Fox, Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. in International Conference on Machine Learning (PMLR, Honolulu, Hawaii, USA, 2023), pp. 26311–26325
- J. Ruan, Y. Chen, B. Zhang, Z. Xu, T. Bao, G. Du, S. Shi, H. Mao, X. Zeng, R. Zhao, Tptu: Task planning and tool usage of large language model-based ai agents. in NeurIPS 2023 Foundation Models for Decision Making Workshop (New Orleans, Louisiana, United States of America, 2023)
- Z. Zhao, E. Wallace, S. Feng, D. Klein, S. Singh, Calibrate before use: Improving few-shot performance of language models. in International conference on machine learning (PMLR, Virtual Event, 2021), pp. 12697–12706
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
- T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, T. Scialom, Toolformer: Language models can teach themselves to use tools. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 68539-68551
- M. Carroll, R. Shah, M.K. Ho, T. Griffiths, S. Seshia, P. Abbeel, A. Dragan, On the utility of learning about humans for human-ai coordination. Adv. Neural Inf. Process. Syst. 32(2019), pp. 5174-5185
- X. Ning, Z. Lin, Z. Zhou, Z. Wang, H. Yang, Y. Wang, Skeleton-of-thought: Prompting llms for efficient parallel generation. in The Twelfth International Conference on Learning Representations (Vienna, Austria, 2024)
- S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K.E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu et al., Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. in Conference on robot learning (PMLR, London, UK, 2021), pp. 477–490
- C. Colas, L. Teodorescu, P.Y. Oudeyer, X. Yuan, M.A. Côté, Augmenting autotelic agents with large language models. in Conference on Lifelong Learning Agents (PMLR, McGill University, Montréal, Québec, Canada, 2023), pp. 205–226
- L. Zhu, L. Hu, J. Lin, S. Han, Lift: Efficient layer-wise fine-tuning for large model models. in Proceedings of the 31st ACM International Conference on -Multimedia (Association for Computing Machinery, New York, NY, United States, 2023), pp. 4678-4687
- N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-efficient transfer learning for nlp. in International conference on machine learning (PMLR, Long Beach, California, USA, 2019), pp. 2790–2799
- T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 10088-10115
- A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, Superglue: A stickier benchmark for general-purpose language understanding systems. Adv. Neural Inf. Process. Syst. 32 (2019). pp. 3266-3280
- A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang et al., Self-refine: Iterative refinement with self-feedback. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 46534-46594
- N. Shinn, B. Labash, A. Gopinath, Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint (2023) arXiv: 2303.11366 · arXiv
- E. Brooks, L. Walls, R.L. Lewis, S. Singh, Large language models can implement policy iteration. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 30349-30366
- T. Carta, C. Romac, T. Wolf, S. Lamprier, O. Sigaud, P.Y. Oudeyer, Grounding large language models in interactive environments with online reinforcement learning. in International Conference on Machine Learing (PMLR, Honolulu, Hawaii, USA, 2023), pp. 3676–3713
- S. Sudhakaran, M. González-Duque, M. Freiberger, C. Glanois, E. Najarro, S. Risi, Mariogpt: Open-ended text2level generation through large language models. Adv. Neural Inf. Process. Syst. 36 (2024). pp. 54213-54227
- F. Cardoso Durier da Silva, R. Vieira, A.C. Garcia, Can machines learn to detect fake news? a survey focused on social media, in 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (Kuala Lumpur, Malaysia, 2021), pp.1-7
- B. Buchanan, A. Lohn, M. Musser, K. Sedova, Truth, lies, and automation. Cent. Secur. Emerg. Technol. 1(1), 2 (2021)
- N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., Extracting training data from large language models. in 30th USENIX Security Symposium (USENIX Security 21) (USENIX, Berkeley, CA, 2021), pp. 2633–2650
- Z. Xi, T. Du, C. Li, R. Pang, S. Ji, J. Chen, F. Ma, T. Wang, Defending pre-trained language models as few-shot learners against backdoor attacks. Adv. Neural Inf. Process. Syst. 36(2024), pp. 32748-32764
- E. Ostrom, Tragedy of the commons. New Palgrave Dictionary Econ. 2, 1–4 (2008)
- T. Sun, Y. Shao, H. Qian, X. Huang, X. Qiu, Black-box tuning for language-model-as-a-service. in International Conference on Machine Learning (PMLR, Baltimore, Maryland, USA, 2022), pp. 20841–20855
Cited by
Liu, S., McCoy, A. B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf008 · Cited by 200
“57 For example, clinicians and patients have diverse information access needs, some needing to analyze text from a knowledge base, others needing to incorporate structured data from an EHR.”
Bandi, A., Kongari, B., Naguru, R., Pasnoor, S., & Vilipala, S. V. (2025) · Future Internet · doi:10.3390/fi17090404 · Cited by 207
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Yigit, Y., Ferrag, M. A., Ghanem, M. C., Sarker, I. H., Maglaras, L. A., Chrysoulas, C., Moradpoor, N., Tihanyi, N., & Janicke, H. (2025) · Sensors · doi:10.3390/s25061666 · Cited by 69
“In that case, specialized AI agents can instantly coordinate across multiple teams: public utilities, government agencies, and private contractors, ensuring everyone has the latest threat intelligence and situational updates [118].”
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
Metadata
- Authors
- Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y.
- Affiliations
- Wuhan University (CN)
- Zhejiang University (CN)
- Corresponding author
- Li, Xinyi
- Wang, Sai
- Zeng, Siqi
- Wu, Yu
- Yang, Yi
- Venue
- Vicinagearth
- Published
- 2024-10-08
- Volume
- 1
- Issue
- 1
- ISSN
- 3005-060X (ISSN-L)
- OpenAlex
- W4403203873
- Semantic Scholar
- fc8ce12d6186ddaa797e2b36d5e8eb7921425308
- Cited by
- 404
- References
- 252
- Citations by year
- 2026: 254 · 2025: 147 · 2024: 2
- FWCI
- 59.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Vicinagearth. (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- multi-agent LLM systems, large language models, multi-agent systems, artificial general intelligence, agent architecture, agent perception, agent interaction, agent evolution, problem solving, world simulation, autonomous systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. https://app.cochat.ai/papers/10.1007/s44336-024-00009-2
- The paper
- Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024). A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth. https://doi.org/10.1007/s44336-024-00009-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.