PreprintCited by 285Multi-Agent Systems and NegotiationArtificial Intelligence ApplicationsSpeech and dialogue systems

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

Open access
Wu, Qingyun, Bansal, Gagan, Zhang, Jieyu, Wu, Yiran, Li, Beibin, Zhu, Erkang, Jiang, Li, Zhang, Xiaoyun +6 more·2023·arXiv

Abstract

AutoGen is an open-source framework that allows developers to build LLM applications via multiple agents that can converse with each other to accomplish tasks. AutoGen agents are customizable, conversable, and can operate in various modes that employ combinations of LLMs, human inputs, and tools. Using AutoGen, developers can also flexibly define agent interaction behaviors. Both natural language and computer code can be used to program flexible conversation patterns for different applications. AutoGen serves as a generic infrastructure to build diverse applications of various complexities and LLM capacities. Empirical studies demonstrate the effectiveness of the framework in many example applications, with domains ranging from mathematics, coding, question answering, operations research, online decision-making, entertainment, etc.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.08155

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multi-task evaluation uses a set of diverse tasks from different domains to evaluate agent generalization capability in open-domain environments.

    “In this method, people use a set of diverse tasks from different domains to evaluate the agent, which can effectively measure the agent generalization capability in open-domain environments [29,80,153,163,165,166,167].”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [153]

    The Figure 4 caption reports evaluations across four applications—math, question answering, ALFWorld, and coding—supporting the use of diverse tasks. It does not say that these evaluations effectively measure agent generalization in open-domain environments; that part of the attributed claim is unsupported. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • The development of medical AI agents can be approached through two primary methodologies: GUI-based frameworks and coding-based development.

    “The development of medical AI agents can be approached through two primary methodologies: graphical user interface (GUI)-based frameworks and coding-based development.110,111,112”

    From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [110]

    The Introduction and §2.2, “Conversation Programming,” describe AutoGen as a framework for developing agents using programming languages, including Python. That supports the coding-based development part, but the paper does not present GUI-based frameworks or establish these as the two primary methodologies for medical AI agents. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • In AutoGen, each agent maintains a local memory, but the framework enforces controlled memory sharing and coordination protocols, allowing problem-solving behaviors that rely on distributed knowledge.

    “Each agent maintains a local memory, but the framework enforces controlled memory sharing and coordination protocols, thus allowing problem-solving behaviors that rely on distributed knowledge.”

    From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [25]

    The Expanded Discussion (section B; page unavailable) says agents can share knowledge to solve problems collectively and that information can be shared between conversations in a controlled manner. The Task description describes memory handled by a Commander agent and shared with other agents, not local memory maintained by every agent. The paper describes configurable conversation patterns rather than a framework-wide requirement that enforces controlled sharing and coordination protocols. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Generative agents exhibit human-like cognitive and metacognitive abilities.

    “These agents exhibit human-like cognitive and metacognitive abilities, including task planning, situational assessment, progress monitoring, and collaborative efforts among agents.”

    From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [53]

    Section 2.1, “Conversable Agents” (page not available), describes implicit state inference, progress-making and feedback use; “Detailed Workflow” describes an assistant generating plans and making action decisions. “B Expanded Discussion” describes agents collaborating. These support the listed activities, but the paper does not establish that the agents have human-like cognitive or metacognitive abilities. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • Agentic AI systems are increasingly deployed in academic and industrial research pipelines to automate multi-stage knowledge compilation.

    “Agentic AI systems are increasingly deployed in academic and industrial research pipelines to automate multi-stage knowledge compilation.”

    From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [100]

    The Retrieval-augmented Assistant section describes a multi-agent workflow that processes documents, retrieves relevant passages, and generates answers. But the paper presents this as an AutoGen application, not evidence that agentic systems are increasingly deployed in academic and industrial research pipelines for knowledge compilation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 32

  1. Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. Eval- uating correctness and faithfulness of instruction-following models for question answering. arXiv preprint arXiv:2307.16877, 2023. · arXiv
  2. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Col- lisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference on human factors in computing systems, 2019.
  3. Carrie J. Cai, Samantha Winter, David F. Steiner, Lauren Wilcox, and Michael Terry. "hello ai": Uncovering the onboarding needs of medical practitioners for human-ai collaborative decision- making. Proceedings of the ACM on Human-Computer Interaction, 2019.
  4. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023. · arXiv
  5. Victor Dibia. LIDA: A tool for automatic generation of grammar-agnostic visualizations and info- graphics using large language models. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 3: System Demonstrations), Toronto, Canada, July 2023. Association for Computational Linguistics.
  6. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. · arXiv
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
  8. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. · arXiv
  9. Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1999.
  10. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491, 2023. · arXiv
  11. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019.
  12. Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end- to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125, 2017. · arXiv
  13. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gen- eration for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 2020.
  14. Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.03875, 2023a. · arXiv
  15. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802, 2018. · arXiv
  16. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. · arXiv
  17. Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inven- tory and discussion. ACM Journal of Data and Information Quality, 2023.
  18. Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. · arXiv
  19. Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601, 2021. · arXiv
  20. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. · arXiv
  21. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/ abs/1908.10084. · arXiv
  22. Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simu- lation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
  23. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning. PMLR, 2017.
  24. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.03768. · arXiv
  25. Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017. · arXiv
  26. Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 2021.
  27. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a. · arXiv
  28. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023b. · arXiv
  29. Daniel S. Weld and Oren Etzioni. The first law of robotics (a call to arms). In AAAI Conference on Artificial Intelligence, 1994.
  30. Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. An empirical study on challenging math problem solving with gpt-4. arXiv preprint arXiv:2306.01337, 2023. · arXiv
  31. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. · arXiv
  32. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. · arXiv

Cited by

Sorted by citations

Authors
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., & Wang, C.
Affiliations
  • Pennsylvania State University (US)
  • Microsoft (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4385965642
Cited by
193
References
0
Citations by year
2026: 61 · 2025: 71 · 2024: 58 · 2023: 2 · 2019: 1
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (0424422)
  • University of Pennsylvania
  • Pennsylvania State University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
AutoGen, compilers, multi-agent systems, large language models, chatbots, LLM applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review