The DOI matches its DataCite record; 85 of the 99 references it cites were found in a registry or an index; 2 of the 6 sources its claims cite were checked for support.
Multi-agent Architecture Search via Agentic Supernet
Open accessAbstract
Large Language Model (LLM)-empowered multi-agent systems extend the cognitive boundaries of individual agents through disciplined collaboration and interaction, while constructing these systems often requires labor-intensive manual designs. Despite the availability of methods to automate the design of agentic workflows, they typically seek to identify a static, complex, one-size-fits-all system, which, however, fails to dynamically allocate inference resources based on the difficulty and domain of each query. To address this challenge, we shift away from the pursuit of a monolithic agentic system, instead optimizing the \textbf{agentic supernet}, a probabilistic and continuous distribution of agentic architectures. We introduce MaAS, an automated framework that samples query-dependent agentic systems from the supernet, delivering high-quality solutions and tailored resource allocation (\textit{e.g.}, LLM calls, tool calls, token cost). Comprehensive evaluation across six benchmarks demonstrates that MaAS \textbf{(I)} requires only $6\sim45\%$ of the inference costs of existing handcrafted or automated multi-agent systems, \textbf{(II)} surpasses them by $0.54\%\sim11.82\%$, and \textbf{(III)} enjoys superior cross-dataset and cross-LLM-backbone transferability.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
MaAS reframes automated multi-agent design as optimizing an “agentic supernet,” a probabilistic distribution over agentic architectures, and samples query-dependent systems at inference. Across six benchmarks it improves accuracy over handcrafted and automated baselines while using substantially fewer tokens and lower cost.
Key findings
- MaAS achieves the best average on GSM8K, MATH, MultiArith, HumanEval, and MBPP: 83.59 average, with gains over baselines of 0.54%–16.89%; e.g., MATH 51.82 vs. AFlow 51.28, HumanEval 92.85 vs. AFlow 90.93, MBPP 82.17 vs. AFlow 81.67.
- On GAIA, MaAS reaches 20.69 average vs. AFlow 8.00, AgentSquare 16.34, and GPTSwarm 16.33; Level 1 25.91, Level 2 22.01, Level 3 6.25.
- On MATH, MaAS uses $3.38 training cost vs. AFlow’s $22.50, $0.42 inference cost vs. AFlow’s $1.66, 53 min training wall-clock vs. AFlow’s 184 min and DyLAN’s 508 min, and 19 min inference wall-clock.
- Ablation: removing the textual gradient drops HumanEval from 92.85 to 90.17 and MATH from 51.82 to 48.23; removing early exit raises HumanEval cost from 1.01 to 1.67 and MATH cost from 0.86 to 1.04 (10^-3 $).
Methods
- Search space: agentic operators such as CoT, LLM-Debate, Self-Consistency, Self-Refine, Ensemble, Testing, ReAct, and early exit; a multi-agent system is a DAG, and the agentic supernet is an L-layer probabilistic distribution over operators. A controller network samples query-dependent architectures using MoE-style activation scores with threshold thres = 0.3 and an early-exit operator.
- Optimization: cost-constrained objective with trade-off λ; distribution π is updated via empirical Bayes Monte Carlo gradient with cost-aware importance weights, while operators are updated via agent-based textual gradient for prompts, temperature, and node structure. Settings include L = 4, K = 4, and λ ∈ {1e-3, 5e-3, 1e-2}.
- Evaluation: six public benchmarks—GSM8K, MATH, MultiArith, HumanEval, MBPP, and GAIA—with a 1:4 train:test split; MATH uses 617 selected problems. Base LLM is gpt-4o-mini, with Qwen-2.5-72b/70b and llama-3.1-70b for transfer tests; baselines include CoT, ComplexCoT, Self-Consistency, MultiPersona, LLM-Debate, LLM-Blender, DyLAN, AgentVerse, MacNet, AutoAgents, GPTSwarm, ADAS, AgentSquare, and AFlow.
Limitations
- Stated limitations: Not stated in the paper; the ethical statement says the framework raises no ethical concerns.
- Evident: the operator repository is manually initialized and customizable, so search quality depends on the chosen operators; the textual gradient component is adapted from ADAS and AgentSquare and relies on LLM-generated textual updates.
- Evident: evaluation covers six benchmarks and a limited set of LLM backbones; the text inconsistently says “six” and “seven” benchmarks and lists SVAMP in the introduction but MultiArith in the experiments, which may affect reproducibility and generalization claims.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
24 Own result · 6 Own interpretation · 4 No basis given · 2 Sources not found · 2 No full text of the sources · 1 Partly supported · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“for instance, in the GAIA benchmark (Mialon et al., 2023), there is no single system that is optimal for both fi le reading and web searching tasks, leaving practitioners with no alternative but to split the benchmark and optimize separately (Zhuge et al., 2024).”
What the sources say
- Partly supportedcites Mialon et al., 2023 sourceExperimental — accuracy not yet measured
In the GPT4 plugins discussion (page unavailable), the paper says evaluators had to choose between a file-reading mode and third-party plugins selected for each task. Figure 5 compares scores by capability, but the text does not establish that no single system is optimal for both file-reading and web-searching tasks. The recommendation to split the benchmark is attributed to Zhuge et al., not this source. (LLM, conf=0.78)
- Could not verifycites Zhuge et al., 2024 sourceExperimental — accuracy not yet measured
Section 3.4 (GAIA; page unavailable) reports overall benchmark accuracy for systems using different tools, including WebSearch and FileAnalyzer. It does not report which system performs best on file-reading tasks versus web-search tasks, so our copy cannot establish that no single system is optimal for both. Searches for that comparison and for separate optimization found no supporting passage. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the more challenging queries (c) and (d), MaAS sampled additional layers”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MaAS achieves an average best score of 83 . 59% , demonstrating its versatility and superiority.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“removing the textual gradient causes the largest performance drop”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the agentic supernet optimized by MaAS transfers well to models such as Qwen-2.5-70b , with 4 . 98% ∼ 5 . 50% ↑ in performance”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Obs. ❶ MaAS achieves optimal performance across all task domains.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“surpassing existing handcrafted or automated multi-agent systems by 0 . 54% ∼ 16 . 89%”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Obs. ❹ MaAS learns to query-aware early exit from the reasoning process.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when faced with the easy queries (a) and (b), MaAS exits multi-agent architecture sampling at the second layer with probabilities of 0 . 37 and 0 . 47 , respectively, selecting the early-exit operator.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“❹ inductive , demonstrating strong generalizability to unseen agentic operators.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 14
- The quote only partly states this claim. 1
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MaAS learns to avoid overly confident early stopping and instead prioritizes testing and self-refinement in deeper layers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“query (b) chose two agentic operators at the first layer: direct I/O and ReAct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DyLAN taking 508 minutes and GPTSwarm taking 129 minutes. In contrast, the optimization wall-clock time of MaAS requires only 53 minutes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The multi-agent system optimized by MaAS outperforms manually designed methods by an average of 3 . 90 ∼ 6 . 40% and existing automated methods by 2 . 07 ∼ 8 . 26% .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MaAS achieves the highest accuracy with an API cost of 0 . 42$ , demonstrating its high performance and token economy.”
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Removing O exit and C ( · ) results in little impact on performance, but it weakens MaAS's query-dependent nature and unnecessarily increases the inference cost.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“❸ transferable across datasets and LLM-backbones”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“among the various optimization-oriented agentic workflows, MaAS achieves the highest accuracy with the least training token consumption.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MaAS has the lowest prompt/completion token consumption, the lowest API cost, and the shortest wall-clock time during inference.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we demonstrate that MaAS is both training/inference cost-efficient from the following three dimensions”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“❷ tokeneconomical , outperforming the SOTA baseline AFlow on the MATH benchmark with 15% of the training cost and 25% of the inference cost”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MaAS can adaptively sample customized agentic systems for different domains, achieving 18 . 38% and 17 . 61% improvements on Level 1 and 2 tasks, respectively.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We respectfully note that the selection of these operators is highly customizable, allowing users the flexibility to incorporate their desired operators into the operator repository of MaAS.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We introduce the early exit operator, which interrupts the multi-agent architecture sampling process and enables the depth of the agentic supernet to be query-dependent.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“LLM-Debate (Du et al., 2023) allows multiple LLMs to debate, leveraging diverse perspectives to identify better solutions.”
What the sources say
- Not matchedcites Du et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than directly outputting an answer.”
What the sources say
- Not checked — no full text of the sourcecites Wei et al., 2022 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Following (Yao et al., 2023), this operator enables the agent to leverage versatile tools, including code interpreter, web searching, external knowledge database, etc. , to handle diverse user demands.”
What the sources say
- Not matchedcites Yao et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach enables fine-grained resource allocation tailored to query difficulty and domain, significantly improving efficiency while maintaining high-quality outputs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Ph.D.-level abstract algebra may indeed require complicated, token-heavy systems, while simple elementary-level arithmetic works well with a single zero-shot I/O.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“This approach enhances its capability to solve complex problems through intermediate reasoning steps, improving task handling and providing greater transparency in the decision-making process.”
What the sources say
- Not checked — no full text of the sourcecites Wei et al., 2022 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MaAS introduces a new paradigm in multi-agent system design by replacing static, one-sizefits-all architectures with a dynamic and adaptive agentic supernet.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
The difficulty and domain of a query can be captured by the controller network from the query embedding to select an appropriate multi-agent architecture.
“The core of MaAS lies in tailoring a customized multi-agent system for each user query, which may vary in difficulty and domain, to deliver a satisfactory solution”
The paper relies on this without support.
Token cost adequately represents the inference cost of a multi-agent system.
“C ( · ) evaluates the cost of multi-agent systems, represented by token cost”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“While AFlow's accuracy is comparable to that of MaAS, its training cost reaches 22 . 50$ , which is 6 . 8 × that of MaAS (merely 3 . 38$ ).”
Reported 6.8 · recomputed 6.656805
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| AFlow (Zhang et al., 2024c) | 91 . 16 ↑ 3 . 71 | 51 . 28 ↑ 4 . 91 | 96 . 22 ↓ 0 . 63 | 90 . 93 ↑ 3 . 85 | 81 . 67 ↑ 9 . 84 | 82 . 25 |”
Reported 4.91 · recomputed 4.99
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1. Introduction · p. 2
“We conduct comprehensive evaluations on seven widely adopted benchmarks, covering diverse use cases in code generation (HumanEval, MBPP), mathematical reasoning (GSM8K, MATH, SVAMP), and diverse tool usage (GAIA).”
§ 4.1. Experiment Setup · p. 5
“Weevaluate MaAS on six public benchmarks covering three domains: (1) math reasoning , GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), and MultiArith (Roy & Roth, 2016)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3. Cost Analysis · p. 7
“Table 3 further highlights that MaAS has the lowest prompt/completion token consumption, the lowest API cost, and the shortest wall-clock time during inference.”
§ 4.3. Cost Analysis · p. 7
“| GPTSwarm | 21 , 325 , 266 | 6 , 369 , 884 | 7 . 02$ | 129 | 3 , 105 , 571 | 788 , 273 | 0 . 93$ | 30 | 47 . 88 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2. Related Work · p. 4
“Similarly, P and T represent prompts and tools. m and n denote the number of LLM-agents and tools invoked in the operator, respectively.”
§ A. Notations · p. 13
“| T | The temperature setting of the LLM. |”
Citations
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“S is given by: ) , where v ( · ) denotes the embedding function using lightweight models like MiniLM (Wang et al., 2020) and Sentence-Bert (Reimers, 2019), and ∥ represents concatenation.”
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705 · cited 1×
“Core NAS techniques, such as reinforcement learning (Zoph, 2016), evolutionary algorithms (Liu et al., 2021), Bayesian optimization (BO) (White et al., 2021), and MCTS (Wang et al., 2021), have inspired analogous approaches in agentic automation, from policy gradient in (Zhuge et al., 2024) to…”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 4×
“Most existing single/multi-agent workflows can be viewed as agentic operators: CoT (Wei et al., 2022) can be considered one with m = 1 and n = 0 , denoted as O CoT ; Self-RAG (Asai et al., 2023) similarly involves m = 1 agent allocation, but is equipped with n = 1 retrieval engine, denoted as O…”
InfluentialSelf-citationDARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · cited 1×
“In NAS, however, these black-box methods were eventually eclipsed by efficient su-pernet training (White et al., 2023), culminating in seminal works like DARTS (Liu et al., 2018) and SNAS (Xie et al., 2018).”
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 1×
“Following Madaan et al. (2023), this operator initially generates an answer using CoT reasoning, then prompts the agent to self-reflect iteratively.”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · cited 4×
“Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimization , such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication , which focuses on orchestrating interact…”
InfluentialAutoML: A survey of the state-of-the-art
He, X. J., Zhao, K., & Chu, X. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,912 · cited 2×
“AutoML.”
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 1,202 · cited 12×
“The agentic supernet can be seen as a cascaded multi-layer workflow, including ❶ multiple agentic operators ( e.g. , CoT (Wei et al., 2022), Multi-agent Debate (Du et al., 2023), ReAct (Yao et al., 2023)), as well as ❷ the parameterized probability distributions of operators across layers.”
InfluentialReflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933 · cited 1×
“We implement the sampling process π ϕ with a Mixture-of-Expert (MoE)-style network (Shazeer et al., 2017; Huang et al., 2024): where S ↓ = sort( S , desc ) , and represents the activation scores of all feasible operators w.r.t. q .”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 911 · cited 4×
“Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimization , such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication , which focuses on orchestrating interact…”
InfluentialA Comprehensive Survey of Neural Architecture Search
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Chen, X., & Wang, X. Y. (2021) · ACM Computing Surveys · doi:10.1145/3447582 · Cited by 865 · cited 3×
“Core NAS techniques, such as reinforcement learning (Zoph, 2016), evolutionary algorithms (Liu et al., 2021), Bayesian optimization (BO) (White et al., 2021), and MCTS (Wang et al., 2021), have inspired analogous approaches in agentic automation, from policy gradient in (Zhuge et al., 2024) to evolutionary search in (…”
Self-citationSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1×
“Adopting the methodology from Wang et al. (2023a), this operator aggregates five CoT reasoning paths and determines the final answer through majority voting.”
Self-citationSNAS: Stochastic Neural Architecture Search
Xie, S., Zheng, H., Liu, C., & Lin, L. (2018) · arXiv · doi:10.48550/arxiv.1812.09926 · Cited by 731 · cited 1×
“In NAS, however, these black-box methods were eventually eclipsed by efficient su-pernet training (White et al., 2023), culminating in seminal works like DARTS (Liu et al., 2018) and SNAS (Xie et al., 2018).”
A Survey on Evolutionary Neural Architecture Search
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Tan, K. C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3100554 · Cited by 728 · cited 1×
“Core NAS techniques, such as reinforcement learning (Zoph, 2016), evolutionary algorithms (Liu et al., 2021), Bayesian optimization (BO) (White et al., 2021), and MCTS (Wang et al., 2021), have inspired analogous approaches in agentic automation, from policy gradient in (Zhuge et al., 2024) to…”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · cited 1×
“S is given by: ) , where v ( · ) denotes the embedding function using lightweight models like MiniLM (Wang et al., 2020) and Sentence-Bert (Reimers, 2019), and ∥ represents concatenation.”
Solving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 5×
“The agentic supernet can be seen as a cascaded multi-layer workflow, including ❶ multiple agentic operators ( e.g. , CoT (Wei et al., 2022), Multi-agent Debate (Du et al., 2023), ReAct (Yao et al., 2023)), as well as ❷ the parameterized probability distributions of operators across layers.”
InfluentialTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Jiang, D., Ren, X., & Lin, B. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.792 · Cited by 506
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · cited 1×
“1-70b ( Dubey et al., 2024)).”
Self-citationHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
AFlow: Automating Agentic Workflow Generation
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10762 · Cited by 359 · cited 7×
“Subsequently, ADAS (Hu et al., 2024a), AgentSqure (Shang et al., 2024), and AFlow (Zhang et al., 2024c) broaden the design search space.”
InfluentialSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 343
AutoAgents: A Framework for Automatic Agent Generation
Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., & Shi, Y.-M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2309.17288 · Cited by 326 · cited 4×
“Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimization , such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication , which focuses on orchestrating interact…”
InfluentialSelf-citationEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.19118 · Cited by 320 · cited 1×
“…(Zhuge et al., 2024) or competitively (Zhao et al., 2023), can surpass the cognitive and intellectual capabilities of individuals (Du et al., 2023; Liang et al., 2023; Wang et al., 2023b; Jiang et al., 2023; Wu et al., 2023; Zhang et al., 2024a), showcasing the collective intelligence in a…”
Self-citationA Dynamic LLM-Powered Agent Network for Task-Oriented Agent CollaborationPreprint
Liu, Z., Zhang, Y., Li, P., Liu, Y., & Yang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02170 · Cited by 320
Camel: Communicative agents for" mind" exploration of large language model society
Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2264 · Cited by 314
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 1×
“However, this is far from an optimal solution: for example, in mathematical benchmarks (Hendrycks et al., 2021), Ph.D.-level abstract algebra may indeed require complicated, token-heavy systems, while simple elementary-level arithmetic works well with a single zero-shot I/O.”
Complexity-Based Prompting for Multi-Step ReasoningPreprint
Fu, Y., Peng, H. M., Sabharwal, A., Clark, P. A., & Khot, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00720 · Cited by 298 · cited 1×
“We compare MaAS with three series of agentic baselines: (1) single agent execution methods , including CoT (Wei et al., 2022), ComplexCoT (Fu et al., 2022 2024) and llama-3.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
White, C., Neiswanger, W., & Savani, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17233 · Cited by 287
Solving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274
MemoryBank: Enhancing Large Language Models with Long-Term Memory
Zhong, W., Guo, L., Gao, Q., Ye, H., & Wang, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10250 · Cited by 253
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08155 · Cited by 193
(2019) · Some Glad Morning · doi:10.2307/j.ctvq4c06f.22 · Cited by 189
Neural Architecture Search: Insights from 1000 PapersPreprint
White, C., Safari, M., Sukthanker, R. S., Ru, B., Elsken, T., Zela, A., Dey, D., & Hutter, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.08727 · Cited by 144 · cited 1×
“In NAS, however, these black-box methods were eventually eclipsed by efficient su-pernet training (White et al., 2023), culminating in seminal works like DARTS (Liu et al., 2018) and SNAS (Xie et al., 2018).”
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Jiang, D., Ren, X., & Lin, B. Y. (2023) · arXiv · doi:10.48550/arxiv.2306.02561 · Cited by 141 · cited 2×
“Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-powered agents from different sources outputting answers to the same query.”
Empirical Bayes: Past, Present and Future
Carlin, B. P., & Louis, T. A. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474331 · Cited by 138 · cited 1×
“Therefore, we employ an empirical Bayes Monte Carlo procedure (Carlin & Louis, 2000; Yan et al., 2021) to estimate the gradient w.r.t the distribution π : where m k denotes the cost-aware importance weights of the agentic architecture.”
Mind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · cited 1×
“…as question answering (Zhu et al., 2024a), data analysis (Hong et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and data synthesis (Butt et al., 2024), by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen…”
(2023) · doi:10.52202/075280-1220 · Cited by 134
G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural NetworksPreprint
Zhang, G., Yue, Y., Sun, X., Wan, G., Yu, M., Fang, J., Wang, K., Chen, T., & Cheng, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.11782 · Cited by 133 · cited 2×
“…the backpropagation for agentic operators, as visualized in Figure 3 and formalized as follows: where T P , T T , T N represent agent-generated gradient analyses in textual format, corresponding to updates of the prompt, model temperature, and operator node structure (such as merging,…”
Self-citationQwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128 · cited 1×
“0.01 0.19 0.06 0.06 0.45 0.13 0.00 0.06 0.12 0.31 0.05 0.05 0.36 0.01 0.06 0.13 0.17 0.02 0.05 0.23 0.34”
InfluentialSymbolic Learning Enables Self-Evolving Agents
Ou, Y., Zhou, W., Ding, S., et al. (2024) · AI Open · doi:10.48550/arxiv.2406.18532 · Cited by 108
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · cited 7×
“…on the success of single agents (Shen et al., 2024; Zhu et al., 2024b; Zhong et al., 2024 substantially enhance individual model capabilities (Wang et al., 2024a), as demonstrated in early attempts such as AutoGen (Wu et al., 2023), LLM-Debate (Du et al., 2023), and AgentVerse (Chen et al., 2023b).”
InfluentialSample-Efficient Neural Architecture Search by Learning Actions for Monte Carlo Tree Search
Wang, L., Xie, S., Li, T., Fonseca, R., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3071343 · Cited by 98 · cited 3×
“Core NAS techniques, such as reinforcement learning (Zoph, 2016), evolutionary algorithms (Liu et al., 2021), Bayesian optimization (BO) (White et al., 2021), and MCTS (Wang et al., 2021), have inspired analogous approaches in agentic automation, from policy gradient in (Zhuge et al., 2024) to evolutionary search in (…”
Harder Tasks Need More Experts: Dynamic Routing in MoE Models
Huang, Q., An, Z., Zhuang, N., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2403.07652 · Cited by 93
AutoKaggle: A Multi-Agent Framework for Autonomous Data Science CompetitionsPreprint
Li, Z., Zang, Q., Ma, D., Guo, J., Zheng, T., Liu, M., Niu, X., Wang, Y., Yang, J., Liu, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20424 · Cited by 89 · cited 1×
“…2023) have made remarkable strides in a spectrum of domains, such as question answering (Zhu et al., 2024a), data analysis (Hong et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and data synthesis (Butt et al., 2024), by equipping LLMs…”
Self-citationWang, Z., Mao, S., Wu, W., Ge, T., Wei, F., & Ji, H. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.15 · Cited by 74
Can Graph Learning Improve Planning in LLM-based Agents?
Wu, X.-X., Shen, Y.-F., Shan, C., et al. (2024) · Neural Information Processing Systems · doi:10.52202/079017-0173 · Cited by 71
Self-Evolving Multi-Agent Collaboration Networks for Software DevelopmentPreprint
Yue, H., Cai, Y., Du, Y., Zhu, X., Liu, X., Yu, Z., Hou, Y., Tang, S., & Chen, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.16946 · Cited by 70 · cited 4×
“To address this, we utilize agent-based textual gradient (Hao et al., 2023; Liu et al., 2023; Hu et al., 2024b; Zhou et al., 2024) to approximate the backpropagation for agentic operators, as visualized in Figure 3 and formalized as follows: where T P , T T , T N represent agent-generated gradient…”
InfluentialMemGPT: Towards LLMs as Operating Systems
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2310.08560 · Cited by 64 · cited 1×
“…by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen et al., 2024), tools (Shen et al., 2024; Richards & et al., 2023), planning (Qiao et al., 2024; Wu et al., 2024; He et al., 2023), and memory (Zhong et al., 2024; Hatalis et al., 2023; Packer et al., 2023).”
Data Interpreter: An LLM Agent For Data Science
Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Zhang, C., Wei, C., Li, D., Chen, J., Zhang, J., et al. (2024) · arXiv · Cited by 60 · cited 2×
“…Nakajima, 2023; Reworkd, 2023) have made remarkable strides in a spectrum of domains, such as question answering (Zhu et al., 2024a), data analysis (Hong et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and data synthesis (Butt et al.,…”
ChatLLM Network: More brains, More intelligencePreprint
Hao, R., Hu, L., Qi, W., et al. (2023) · AI Open · doi:10.48550/arxiv.2304.12998 · Cited by 58
EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., & Yang, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.08532 · Cited by 50 · cited 4×
“Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimization , such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication , which focuses on orchestrating interact…”
InfluentialMemory Matters: The Need to Improve Long-Term Memory in LLM-Agents
Hatalis, K., Christou, D., Myers, J., Jones, S., Lambert, K., Amos-Binks, A., Dannenhauer, Z., & Dannenhauer, D. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27688 · Cited by 42 · cited 1×
“…by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen et al., 2024), tools (Shen et al., 2024; Richards & et al., 2023), planning (Qiao et al., 2024; Wu et al., 2024; He et al., 2023), and memory (Zhong et al., 2024; Hatalis et al., 2023; Packer et al., 2023).”
AutoTQA: Towards Autonomous Tabular Question Answering through Multi-Agent Large Language Models
Zhu, J.-P., Cai, P., Xu, K., Li, L., Sun, Y., Zhou, S., Su, H., Liu, T., & Liu, Q. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3685800.3685816 · Cited by 41 · cited 1×
“…agents (Richards & et al., 2023; Nakajima, 2023; Reworkd, 2023) have made remarkable strides in a spectrum of domains, such as question answering (Zhu et al., 2024a), data analysis (Hong et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and…”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.10848 · Cited by 39
KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents
Zhu, Y., Qiao, S., Ou, Y., Deng, S., Lyu, S., Shen, Y., Liang, L., Gu, J., Chen, H., & Zhang, N. (2024) · arXiv · doi:10.48550/arxiv.2403.03101 · Cited by 38
CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents
Zhao, Q., Wang, J., Zhang, Y., Jin, Y., Zhu, K., Chen, H., & Xie, X. (2023) · arXiv · doi:10.48550/arxiv.2310.17512 · Cited by 37
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
Huang, D., Zhang, J. M., Luck, M., Bu, Q., Qing, Y., & Cui, H. (2023) · arXiv · doi:10.48550/arxiv.2312.13010 · Cited by 34 · cited 1×
“Following the test designer in AgentCoder (Huang et al., 2023), this operator is used for generating test cases for the generated code.”
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2309.16797 · Cited by 31 · cited 2×
“Efforts to automate the design of agent-based systems can be broadly classified into the following categories: (I) Prompt Optimization , such as PromptBreeder (Fernando et al., 2023), DsPy (Khattab et al., 2023), and EvoPrompt (Guo et al., 2023); (II) Inter-agent Communication , which focuses on orchestrating interact…”
FP-NAS: Fast Probabilistic Neural Architecture Search
Yan, Z., Dai, X., Zhang, P., Tian, Y., Wu, B., & Feiszli, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01489 · Cited by 24 · cited 1×
“Therefore, we employ an empirical Bayes Monte Carlo procedure (Carlin & Louis, 2000; Yan et al., 2021) to estimate the gradient w.r.t the distribution π : where m k denotes the cost-aware importance weights of the agentic architecture.”
Sibyl: Simple yet Effective Agent Framework for Complex Real-world ReasoningPreprint
Wang, Y., Shen, T., Liu, L., & Xie, J. G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.10718 · Cited by 21 · cited 1×
“Table 2 shows a comparison of MaAS with automated systems and three additional baselines, including Auto-GPT (Richards & et al., 2023), TapeAgent (Bahdanau et al., 2024), and Sibyl (Wang et al., 2024b) on the GAIA benchmark.”
BENCHAGENTS: Automated Benchmark Creation with Agent Interaction
Butt, N., Chandrasekaran, V., Joshi, N., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.22584 · Cited by 20 · cited 1×
“…data analysis (Hong et al., 2024; Li et al., 2024), code generation (Shinn et al., 2023), web navigation (Deng et al., 2024), and data synthesis (Butt et al., 2024), by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen et al., 2024), tools (Shen et al.,…”
From Persona to Personalization: A Survey on Role-Playing Language Agents
Chen, J., Wang, X., Xu, R., Yuan, S., Zhang, Y., Shi, W., Xie, J., Li, S., Yang, R., Zhu, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18231 · Cited by 15 · cited 6×
“…DsPy (Khattab et al., 2023) and Evo-Prompting (Guo et al., 2023) automate prompt optimization, GPTSwarm (Zhuge et al., 2024) and G-Designer (Zhang et al., 2024b) optimize inter-agent communication, and EvoAgent (Yuan et al., 2024) and AutoAgents (Chen et al., 2023a) self-evolve agent profiling.”
InfluentialSelf-citationHe, Z.-T., Cao, P., Chen, Y., Liu, K., Li, R., Sun, M., & Zhao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.613 · Cited by 14 · cited 6×
“…by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen et al., 2024), tools (Shen et al., 2024; Richards & et al., 2023), planning (Qiao et al., 2024; Wu et al., 2024; He et al., 2023), and memory (Zhong et al., 2024; Hatalis et al., 2023; Packer et al., 2023).”
InfluentialCooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents
Piatti, G., Jin, Z., Kleiman-Weiner, M., Schölkopf, B., Sachan, M., & Mihalcea, R. (2024) · arXiv · doi:10.48550/arxiv.2404.16698 · Cited by 14 · cited 3×
“Similarly, P and T represent prompts and tools. m and n denote the number of LLM-agents and tools invoked in the operator, respectively.”
GAIA: a benchmark for General AI Assistants
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2311.12983 · Cited by 12 · cited 1×
“…paradigm becomes even more problematic when applied to benchmarks across multiple task domains ( Dilemma 2 ): for instance, in the GAIA benchmark (Mialon et al., 2023), there is no single sys-tem that is optimal for both file reading and web searching tasks, leaving practitioners with no…”
VerifAI: Verified Generative AI
Tang, N., Yang, C., Fan, J., Cao, L., Luo, Y., & Halevy, A. (2023) · arXiv · doi:10.48550/arxiv.2307.02796 · Cited by 12 · cited 1×
“This dependency significantly limits the rapid adaptation of multi-agent systems to diverse domains and application scenarios (Tang et al., 2023; Zhang et al., 2024c).”
TapeAgents: a Holistic Framework for Agent Development and OptimizationPreprint
Bahdanau, D., Gontier, N., Huang, G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.08445 · Cited by 12 · cited 1×
“Table 2 shows a comparison of MaAS with automated systems and three additional baselines, including Auto-GPT (Richards & et al., 2023), TapeAgent (Bahdanau et al., 2024), and Sibyl (Wang et al., 2024b) on the GAIA benchmark.”
Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., & Ji, H. (2023) · arXiv · doi:10.48550/arxiv.2307.05300 · Cited by 11 · cited 2×
“…2023), web navigation (Deng et al., 2024), and data synthesis (Butt et al., 2024), by equipping LLMs with high-* level features, including persona (Wang et al., 2023b; Chen et al., 2024), tools (Shen et al., 2024; Richards & et al., 2023), planning (Qiao et al., 2024; Wu et al., 2024; He et al.,…”
Automated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10 · cited 4×
“Subsequently, ADAS (Hu et al., 2024a), AgentSqure (Shang et al., 2024), and AFlow (Zhang et al., 2024c) broaden the design search space.”
InfluentialScaling Large Language Model-based Multi-Agent Collaboration
Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07155 · Cited by 8 · cited 1×
“…systems, such as CAMEL (Li et al., 2023), AutoGen (Wu et al., 2023), and MetaGPT (Hong et al., 2023), while delivering specialized capacity, often heavily rely on manual configurations, including prompt engineering, agent profiling, and inter-agent communication pipelines (Qian et al., 2024).”
Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems
Zhang, G., Yue, Y., Li, Z., Yun, S., Wan, G., Wang, K., Cheng, D., Yu, J. X., & Chen, T. (2024) · arXiv · doi:10.48550/arxiv.2410.02506 · Cited by 4
Self-citationAgentSquare: Automatic LLM Agent Search in Modular Design Space
Shang, Y., Li, Y., Zhao, K., Ma, L., Liu, J., Xu, F., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2410.06153 · Cited by 3 · cited 5×
“Subsequently, ADAS (Hu et al., 2024a), AgentSqure (Shang et al., 2024), and AFlow (Zhang et al., 2024c) broaden the design search space.”
InfluentialAutoAct: Automatic Agent Learning from Scratch for QA via Self-PlanningPreprint
Qiao, S., Zhang, N., Fang, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.05268 · Cited by 3
Language Agents as Optimizable Graphs
Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., & Schmidhuber, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16823 · Cited by 3
GraphRouter: A Graph-based Router for LLM Selections
Feng, T., Shen, Y., & You, J. (2024) · arXiv · doi:10.48550/arxiv.2410.03834 · Cited by 2 · cited 2×
“Building on the success of single agents (Shen et al., 2024; Zhu et al., 2024b; Zhong et al., 2024 substantially enhance individual model capabilities (Wang et al., 2024a), as demonstrated in early attempts such as AutoGen (Wu et al., 2023), LLM-Debate (Du et al., 2023), and AgentVerse (Chen et…”
Laird, N. M. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2379.pub2 · Cited by 1
EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms
Yuan, S., Song, K., Chen, J., Tan, X., Li, D., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14228
Archon: An Architecture Search Framework for Inference-Time Techniques
Saad-Falcon, J., Lafuente, A. G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E. K., Chen, M., Guha, N., Ré, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.15254
Research works printed but not linked: 1
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Petroski Such, F., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Hebgen Guss, W., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, July 01, 2021 2021.
Cited by
Sorted by citations
Large Language Models Meet NLP: A Survey
Qin, L.-B., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y.-H., Li, M., Che, W.-X., & Yu, P. S. (2024) · Frontiers Comput. Sci. · doi:10.48550/arxiv.2405.12819 · Cited by 191
AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?Preprint
Zhang, G.-M., Wang, J., Chen, J., Zhou, W., Wang, K., & Yan, S.-C. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2509.03312 · Cited by 145
“1, CRITIC consistently degrades performance ( e.g. , CRITIC+MaAS+GAIA accuracy drops by − 4 .”
Self-citationOn the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Huang, J.-T., Zhou, J., Jin, T., Zhou, X., Chen, Z., Wang, W., Yuan, Y., Lyu, M. R., & Sap, M. (2024) · arXiv · doi:10.48550/arxiv.2408.00989 · Cited by 1
G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems
Zhang, G.-M., Fu, M., Wan, G.-C., Yu, M., Wang, K., & Yan, S.-C. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.07398 · Cited by 124
“More recent automated MASs, such as GPTSwarm [47], ADAS [48], AFlow [49], and MaAS [50] have made it to automatically optimize inter-agent topologies or prompts, which, nevertheless, ultimately yield giant and cumbersome MAS architectures, lacking the agility to self-adjust with accumulated…”
Self-citationMemEvolve: Meta-Evolution of Agent Memory SystemsPreprint
Zhang, G., Ren, H., Zhan, C., Zhou, Z., Wang, J., Zhu, H., Zhou, W., & Yan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.18746 · Cited by 108
“Storage can be vector databases (Zhao et al., 2024), knowledge graphs (Zhang et al., 2025b; Rasmussen et al., 2025), or others.”
InfluentialSelf-citation
Metadata
- Authors
- Zhang, G., Niu, L., Fang, J., Wang, K., Bai, L., & Wang, X.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2502.04180
- OpenAlex
- W4407247211
- Cited by
- 0
- References
- 99
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- multi-agent systems, architecture search, large language models, tool calls, inference cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-24, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-24 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
