Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies

Similar trends are observed in the test-output-prediction subtask of LiveCodeBench.

“We again observe similar trends in the test-output-prediction subtask of LiveCodeBench (Jain et al., 2024).”

• cites Jain et al., 2024 sourceSupported

Agents in debate can elicit more truthful predictions than single-agent prediction.

“Agents in debate can elicit more truthful predictions than single-agent prediction (Du et al., 2024;Liang et al., 2024)”

• cites Du et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Liang et al., 2024 sourceSupported

This type of multi-agent system typically outperforms its single-agent counterpart by involving more diverse agentic perspectives or role profiles.

“This type of multi-agent system (MAS) typically outperforms its singleagent counterpart by involving more diverse agentic perspectives or role profiles, such as agents as verifiers (Shinn et al., 2024) and multi-agent debate (Qian et al., 2024;Wang et al., 2024b).”

• cites Shinn et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Qian et al., 2024 sourcePartly supported

  The Introduction (p. 2) says task-oriented multi-agent collaboration often surpasses standalone intelligence, and Network Construction (p. 3) describes distinct critic and actor roles. The paper studies collaborative critique and refinement, however; it does not identify MACNET as a multi-agent debate system, the specific example for which this citation is offered. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Wang et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AFlow's performance is attributable to its expansion phase generating new nodes based on an error log contrasting predictions with ground truth, providing implicit textual gradients to reflect on formatting errors in prompt design.

“We attribute the performance of AFlow to: 1) its 'expansion' phase that generates new nodes based on an error log that contrasts the predictions with the ground truth, which provides implicit textual gradients (Pryzant et al., 2023) to reflect on any formatting errors in prompt design;”

• cites Pryzant et al., 2023 sourceSupported

Mass-optimized multi-agent systems outperform existing manually-crafted MAS baselines and automatically-generated alternatives by a substantial margin across reasoning, multi-hop understanding, and code generation tasks.

“We show that M as s-optimized multi-agent systems outperform a spectrum of existing alternatives by a substantial margin.”

• Also stated in: § 1. Introduction · boosted

  “By optimizing over the identified influential components, M as s yields optimized MAS that achieves state-of-the-art performance, outperforming existing manually-crafted MAS baselines and automatically-generated alternatives, by a substantial margin, demonstrated across an extensive selection of tasks, including reasoning, multi-hop understanding, and code generation.”

Prompts frequently form an influential design component that yields strong-performing multi-agent systems.

“Our analyses reveal that prompts frequently form an influential design component that yields strong-performing MAS, and influential topologies only represent a small fraction of the full search space.”

Mass shows consistent improvements on Claude 3.5 Sonnet as reported in Table 4.

“M as s leads to high-performing MAS: 78.8% and 74.3% on average on Gemini 1.5 Pro and Flash, respectively, where we observe consistent improvements on Claude 3.5 Sonnet as reported in Table 4.”

ADAS only brings subtle gains even when conditioning its meta-agent generation on common forms of agents.

“By comparing Mass with state-of-the-art automatic agent design baselines, ADAS and AFlow, we first notice that ADAS only brings subtle gains even by already conditioning its metaagent generation based on the common forms of agents.”

Not all topologies have a positive influence on multi-agent system design.

“We observe that not all topologies have a positive influence on the MAS design.”

Mass's optimization trajectory demonstrates a steady trend that gradually improves validation performance via interleaving search towards better prompts and topologies.

“Mass's trajectory demonstrates a steady trend of optimization that gradually improves the validation performance via interleaving the search towards better prompts and topologies.”

Prompting, which equips agents with more informative instructions and exemplars, demonstrates significant advantages in token-effectiveness over other building blocks such as self-consistency, self-refine, and multi-agent debate.

“In Fig. 2, prompting, which equips agents with more informative instructions and exemplars, demonstrates significant advantages in its token-effectiveness over other building blocks.”

Mass yields substantial gains over common forms of multi-agent systems (self-consistency, self-refine, and multi-agent debate) that scale up without optimizing prompts for agents in collaboration.

“M as s yields substantial gains over common forms of multi-agent systems, (e.g.selfconsistency, self-refine, and multi-agent debate), that scale up without optimizing prompts for agents in collaboration.”

• AssumptionRests on an assumption: The proposed topology search space covers the vast majority of effective multi-agent system designs.

Positively influenced topologies only represent a small fraction of the overall set.

“It is noticeable that not all topologies are beneficial to MAS design, whereas positively influenced topologies only represent a small fraction of the overall set, such that, in HotpotQA (Yang et al., 2018), only debate brings 3% gain while others fail to improve or even degrade systematic performance.”

Mass surpasses prior arts that focused solely on optimizing workflow topology without appropriate prompt designs.

“Building on these, we propose a multistage optimization algorithm, Multi-Agent System Search (M as s), that surpasses prior arts that focused solely on optimizing workflow topology without appropriate prompt designs.”

Going from Stage (1) to Stage (2), another 3% gain can be achieved by composing influential topologies while searching the optimal configurations.

“In addition, going from Stage (1) to ( 2), another 3% gain can be achieved by composing influential topologies while searching the optimal configurations.”

Our approach demonstrates greater effectiveness of MAS design with properly optimized prompts and thoughtfully designed search spaces.

“Instead, our approach demonstrates the greater effectiveness of MAS design with properly optimized prompts and thoughtfully designed search spaces.”

Workflow-level prompt optimization often yields practical benefits.

“Our experiments (Fig. 5 & 6) demonstrate that this stage often yields practical benefits.”

Influential topologies only represent a small fraction of the full search space.

“Our analyses reveal that prompts frequently form an influential design component that yields strong-performing MAS, and influential topologies only represent a small fraction of the full search space.”

Mass obtains further gains of approximately 2% by conducting workflow-level prompt optimization on the best-found topology.

“Lastly, M as s obtains further gains (∼2%) by conducting workflow-level prompt optimization on the best-found topology, which indicates that optimizing the prompts towards modeling the interdependence of agents is beneficial in the MAS design.”

Prompts together with topologies play critical roles in enabling more effective multi-agent system design.

“We reveal that prompts together with topologies play critical roles in enabling more effective MAS design.”

AFlow demonstrates competitive performance to Mass, especially on 2WikiMQA and HumanEval.

“AFlow, on the other hand, demonstrates a competitive performance to M as s, especially on 2WikiMQA and HumanEval.”

Practical gains can be obtained by further conducting workflow-level prompt optimization (3PO) on the best-found topology.

“Practical gains can be obtained by further conducting workflow-level prompt optimization (3PO) on the best-found topology.”

AFlow is exposed to a larger variance in its optimization due to the nature of MCTS.

“However, when it comes to automatic design baselines without explicit prompt optimization stages, AFlow is exposed to a larger variance in its optimization due to the nature of MCTS, whereas ADAS gets trapped in discovering over-complex topologies that appear to be less effective than the prompt design space.”

Mass has brought substantial improvements over a single agent design by searching in a customizable topology space.

“Mass has brought substantial improvements over a single agent design by searching in a customizable topology space.”

• AssumptionRests on an assumption: The proposed topology search space covers the vast majority of effective multi-agent system designs.

When transferring the prompt template for each agent from Gemini to Claude, the basic topology on some tasks may result in severe degradation of performance.

“As we transfer the prompt template for each agent from Gemini to Claude, it is noticeable that the basic topology on some tasks may result in severe degradation of performance, and Mass successfully recovers the performance and brings significant improvements over the initial agent.”

Mass demonstrates advanced token-effectiveness.

“Following Sec.2.1, Mas s also demonstrated advanced token-effectiveness, which we refer to Fig. 9.”

Workflow-level prompt optimization leads to the best-performing predictor for aggregation.

“Workflow-level prompt optimization then leads to the best-performing predictor for aggregation.”

The meta-prompt of AFlow does not generalize to other LLM backbones.

“However, we find the meta-prompt of AFlow does not generalize to other LLM backbones.”

As found in Stage (2), aggregating with more parallel agents outweighs the multi-agent debate.

“However, as found in Stage (2), aggregating with more parallel agents actually outweighs the multi-agent debate.”

ADAS gets trapped in discovering over-complex topologies that appear less effective than the prompt design space.

“However, when it comes to automatic design baselines without explicit prompt optimization stages, AFlow is exposed to a larger variance in its optimization due to the nature of MCTS, whereas ADAS gets trapped in discovering over-complex topologies that appear to be less effective than the prompt design space.”

A single agent might suffer from prompt sensitivity, where simple modifications in the prompt can exert significant but unexpected degradation of performance.

“First, the single agent might suffer from prompt sensitivity (Verma et al., 2024), where simple modifications in the prompt can already exert significant but unexpected degradation of performance (Liu et al., 2024a;Zhou et al., 2024b).”

• cites Verma et al., 2024 sourceNot checked — no full text of the source
• cites Liu et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhou et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The importance of proper prompt designs has been relatively under-studied in prior works such as ADAS and AFlow.

“However, we observe that the importance of proper prompt designs has been relatively under-studied in these prior works.”

Optimizing prompts towards modeling the interdependence of agents is beneficial in MAS design.

“Lastly, M as s obtains further gains (∼2%) by conducting workflow-level prompt optimization on the best-found topology, which indicates that optimizing the prompts towards modeling the interdependence of agents is beneficial in the MAS design.”

AFlow's performance is attributable to a more refined search space within a pre-defined set of operators.

“2) a more refined search space within a pre-defined set of operators.”

The comparison between Mass and AFlow is not completely fair.

“Therefore, the comparison is not completely fair, and we treat the results from AFlow as a good reference.”

Mass-optimized MAS significantly outperforms existing manual and automated approaches across an extensive set of tasks.

“Our experiments demonstrate that Mass-optimized MAS significantly outperforms existing manual and automated approaches across an extensive set of tasks.”

• AssumptionRests on an assumption: The proposed topology search space covers the vast majority of effective multi-agent system designs.