AgentSquare: Automatic LLM Agent Search in Modular Design Space

Probably not a substantial claim: may not pass S3 (load-bearing).

Many experts in the field have proposed building LLM agentic systems with key modular components from engineering and cognitive perspectives.

“Many experts in the field have proposed building LLM agentic systems with key modular components from engineering (Weng, 2023) and cognitive perspectives (Sumers et al., 2023).”

• cites Weng, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sumers et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Another line of research focuses on optimizing the configuration of multi-agent systems.

“Another line of research focuses on optimizing the configuration of multi-agent systems (Chen et al., 2023;Yuan et al., 2024;Li et al., 2023;Zhuge et al., 2024;Wang et al., 2023b).”

• cites Chen et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Yuan et al., 2024 sourceSupported
• cites Li et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The design of module evolution is inspired by the iterative pipeline of FunSearch, which prompts LLMs to propose new solutions based on the target problem and performance feedback from existing solutions.

“This design is inspired by the iterative pipeline of FunSearch (Romera-Paredes et al., 2024), which prompts LLMs to propose new solutions based on the target problem and performance feedback from existing solutions.”

• cites Romera-Paredes et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs have exhibited remarkable reasoning abilities under advanced prompting approaches such as CoT, ToT, and SoT, shaping the foundation of the intelligence of LLM agents.

“LLMs have exhibited remarkable reasoning abilities under advanced prompting approaches such as CoT (Wei et al., 2022), ToT (Yao et al., 2024), and SoT (Shang et al., 2024), shaping the foundation of the intelligence of LLM agents.”

• cites Wei et al., 2022 sourceNot checked — no full text of the source
• cites Yao et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shang et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CoALA is a conceptual framework that tries to provide a unified design principle of LLM agents but remains a vision of how LLM agents should be in the future, without providing a practical design framework.

“Besides, there have been some conceptual frameworks trying to provide a unified design principle of LLM agents, such as CoALA (Sumers et al., 2023).However, it's still a vision of how LLM agents should be in the future, without providing a practical design framework.”

• cites Sumers et al., 2023 sourcePartly supported

  The Introduction and Section 4 describe CoALA as a conceptual framework for organizing existing language agents and guiding the design of new ones. But Section 6, “Actionable Insights,” says how to use its components to develop an application-specific agent and determine a high-level architecture. Calling it only a future vision without practical design guidance overstates the limitation. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Empirical results demonstrate that the predicted performance of agents closely matches their actual performance, verifying the effectiveness of the performance predictor.

“Empirical results demonstrate that the predicted performance of agents closely matches their actual performance, verifying the effectiveness of the proposed performance predictor, which is detailed in Section 4.3.”

Classified as probably not a substantial claim.

The core of the work is a modular design space for LLM agents, comprising 4 categories of modules: Planning, Reasoning, Tool Use, and Memory.

“Therefore, the core of our work is a modular design space for LLM agents, comprising 4 categories of modules: Planning, Reasoning, Tool Use, and Memory.”

May combine two claims.

Evaluating a simple CoT agent based on GPT-4o in ALFWorld requires around $60, making the agent search economically unsustainable at scale.

“For instance, evaluating a simple CoT agent based on GPT-4o in ALFWorld (Shridhar et al., 2021) requires around $60, making the agent search economically unsustainable at scale.”

• cites Shridhar et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The authors sort out 16 popular LLM agents and abstract a modular design space with 1050 possible combinations, which can be easily extended when new modules are discovered.

“As a result, we sort out 16 popular LLM agents and abstract a modular design space with 1050 possible combinations, which can be easily extended when new modules are discovered.”

May combine two claims.

AgentSquare discovers novel LLM agents that outperform all known human-designed agents across six benchmarks, with an average performance gain of 17.2% against best-known human designs.

“Extensive experiments across six benchmarks, covering the diverse scenarios of web, embodied, tool use and game applications, show that AgentSquare substantially outperforms hand-crafted agents, achieving an average performance gain of 17.2% against best-known human designs.”

• Also stated in: § 1 INTRODUCTION · plain

  “Our experiments show AgentSqaure can discover novel LLM agents that outperform hand-crafted agents across all six benchmarks, scoring an average performance gain of 17.2% compared to the best known human designs.”

May combine two claims. The quote only partly states this claim.

Based on GPT-4o, AgentSquare achieves 0.607 on Webshop, 0.695 on ALFWorld, 0.781 on SciWorld, 0.524 on M3Tool, 0.583 on TravelPlanner and 0.669 on PDDL, the highest among all compared methods on all six tasks.

“AgentSquare 0.607 0.695 0.781 0.524 0.583 0.669”

May combine two claims.

Compared with the best human-designed agent, AgentSquare achieves an average 14.1% performance improvement on Webshop, 26.1% improvement on ALFWorld, 20.5% improvement on SciWorld, 30.6% improvement on M3Tool, 6.0% improvement on Travelplanner, 6.0% improvement on PDDL.

“Specifically, as shown in Table 1 and Table A.3, compared with the best human-designed agent, AgentSquare achieves an average 14.1% performance improvement on Webshop, 26.1% improvement on ALFWorld, 20.5% improvement on SciWorld, 30.6% improvement on M3Tool, 6.0% improvement on Travelplanner, 6.0% improvement on PDDL.”

May combine two claims.

The best agent from AgentSquare is commonly cost-efficient, striking the best performance-cost trade-off among all compared agents.

“Simultaneously, the best agent from AgentSquare is commonly cost-efficient, which strikes the best performance-cost trade-off among all compared agents as seen in Figure A.7 -Figure A.12.”

May combine two claims.

The predicted performance closely aligns with the actual performance, demonstrating the effectiveness of the performance predictor.

“It can be found that the predicted performance closely aligns with the actual performance, demonstrating the effectiveness of the performance predictor.”

May combine two claims.

Dropping each design (module evolution or module recombination) results in a noticeable performance decline, and module recombination has a larger impact.

“It can be seen that dropping each design results in a noticeable performance decline and the module recombination has a larger impact.”

May combine two claims.

Compared with the best agent crafted from other search methods, AgentSquare achieves an average improvement of 8.4% on Webshop, 8.1% on ALFWorld, 11.0% on SciWorld, 12.8% on M3Tool, 2.5% on Travelplanner and 1.4% on PDDL.

“AgentSquare achieves an average 8.4% performance improvement on Webshop, 8.1% improvement on ALFWorld, 11.0% improvement on SciWorld, 12.8% improvement on M3Tool, 2.5% improvement on Travelplanner, 1.4% improvement on PDDL.”

May combine two claims.

On GPT-3.5, removing module evolution or removing module recombination lowers AgentSquare's performance on all six tasks relative to the full AgentSquare.

“w/o module evolution 0.595 0.623 0.288 0.236 0.483 0.202 w/o module recombination 0.578 0.546 0.310 0.258 0.267 0.173”

May combine two claims. The quote only partly states this claim.

Based on GPT-3.5, AgentSquare achieves 0.617 on Webshop, 0.651 on ALFWorld, 0.432 on SciWorld, 0.285 on M3Tool, 0.520 on TravelPlanner and 0.219 on PDDL, the highest among all compared methods on all six tasks.

“AgentSquare 0.617 0.651 0.432 0.285 0.520 0.219”

May combine two claims.

AgentSquare demonstrates a steady convergence trajectory, with more advanced agents continually emerging during search.

“AgentSquare demonstrates a steady convergence trajectory, where more advanced agents are continually emerging during search.”

May combine two claims.

The average API cost per search iteration for AgentSquare using GPT-3.5 is $3.16 on Webshop, $4.25 on ALFWorld, $1.92 on SciWorld, $2.43 on M3Tool, $1.84 on TravelPlanner and $2.70 on PDDL.

“Avg cost (GPT-3.5)$3.16 $4.25 $1.92 $2.43 $1.84 $2.70”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are several recent studies that explore the problem of prompting LLMs to design LLM agentic systems.

“There are several recent studies that explore the problem of prompting LLMs to design LLM agentic systems.”

Probably not a substantial claim: may not pass S4 (based).

The high costs of agent evaluation during the search process constrain the overall search scale.

“(3) the high costs of agent evaluation during the search process constrain the overall search scale.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Current LLM workflow program frameworks (e.g., LangChain and Auto-GPT) only provide operation-level components, which cannot support module-level search that best exploits the potential of prior successful designs.

“Besides, current LLM workflow program frameworks (e.g., LangChain and Auto-GPT) only provide operation-level components, which cannot support module-level search that best exploits the potential of prior successful designs.”

May combine two claims.

An in-context surrogate model requires significantly fewer tokens than real environment evaluation, making it more cost-efficient and supporting larger-scale searches.

“Compared to real environment evaluation, such an in-context surrogate model requires significantly fewer tokens, making it more cost-efficient and supporting larger-scale searches.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OPRO and Promptbreeder can be viewed as leveraging the reasoning power of LLMs to improve the prompt of LLM agents.

“OPRO (Yang et al., 2024) and Promptbreeder (Fernando et al., 2024) can be viewed as leveraging the reasoning power of LLMs to improve the prompt of LLM agents.”

• cites Yang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fernando et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Module evolution combined with module recombination enables the search of any module combination in the open-ended solution space.

“This operation, combined with module recombination, enables the search of any module combination in the open-ended solution space.”

May combine two claims.

Searching only with module recombination narrows the solution space and limits the upper bound of agent performance.

“Consequently, searching only with module recombination narrows the solution space and limits the upper bound of agent performance.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The main difference and contribution of this work lie in introducing a modular design space for LLM agents, which can provide a standard framework to support the convenient reuse of existing successful agent components and fruitful innovative agent module discovery.

“Our main difference and contribution lie in introducing a modular design space for LLM agents, which can provide a standard framework to support the convenient reuse of existing successful agent components and fruitful innovative agent module discovery.”

Classified as probably not a substantial claim.

The search space of MoLAS, defined as the Cartesian product of four orthogonal modules, is vast and hard to explore.

“Solving the optimization problem of MoLAS features three key challenges: (1) The search space, defined as the Cartesian product of four orthogonal modules, is vast and hard to explore;”

Classified as probably not a substantial claim. May combine two claims.

AgentSquare is implemented and experiments are conducted using both GPT-3.5-turbo-0125 and GPT-4o.

“We implement AgentSquare and conduct experiments using both GPT-3.5turbo-0125 and GPT-4o (Achiam et al., 2023).”

• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A few recent studies have explored using LLMs to rewrite and optimize the prompts of existing agents.

“A few recent studies have explored using LLMs to rewrite and optimize the prompts of existing agents (Fernando et al., 2024; Yang et al., 2024).”

• cites Fernando et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Yang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Efforts to optimize multi-agent systems are orthogonal to the optimization of single-agent systems, as they focus more on role-playing and interaction patterns among multiple agents, rather than the design of agentic modules.

“These efforts are orthogonal to the optimization of single-agent systems, as they focus more on the role-playing and interaction patterns among multiple agents, rather than the design of agentic modules.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MoLAS is a guided and constrained searching problem in the modular design space, which is a subset of the entire code search proposed in ADAS.

“MoLAS is a guided and constrained searching problem in the modular design space, which is a subset of the entire code search proposed in ADAS (Hu et al., 2024).”

• cites Hu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

These proposals remain largely conceptual, lacking implementable solutions to unify existing LLM agents.

“However, these proposals remain largely conceptual, lacking implementable solutions to unify existing LLM agents.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The module sets in MoLAS encompass any code with standard IO interfaces, making module selection an open-ended problem.

“(2) the module sets encompass any code with standard IO interfaces, making the module selection an openended problem;”

Probably not a substantial claim: may not pass S4 (based).

The solution space for each module type is open-ended, allowing any code with a standardized I/O interface.

“As mentioned above, the solution space for each module type is open-ended, allowing any code with a standardized I/O interface.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A modular design space for LLM agents facilitates the reuse of prior successful designs and supports the exploration of new architectures.

“A modular design space for LLM agents facilitates the reuse of prior successful designs and supports the exploration of new architectures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Such decomposition is critical for handling very complex tasks with long-term characteristics, especially for agents in open-world environments such as MineCraft.

“Such decomposition is critical for handling very complex tasks with long-term characteristics, especially for agents in open-world environments such as MineCraft (Wang et al., 2024a;c).”

• cites Wang et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wang et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

At the core of such modularization is the standardization of input-output interfaces, which ensures both extensibility and seamless integration with existing designs.

“At the core of such modularization is the standardization of input-output interfaces, which ensures both extensibility and seamless integration with existing designs.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The proposed modular design space enables researchers to easily build on previous successful designs and accumulate new discoveries as a community.

“We propose a novel modular design space for LLM agents, enabling researchers to easily build on previous successful designs and accumulate new discoveries as a community.”

Classified as probably not a substantial claim.

The design space is abstracted from a thorough literature review of existing agentic systems.

“This design space is abstracted from a thorough literature review of existing agentic systems (details provided in Section 2).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ADAS introduces the idea of searching the entire agentic system defined in code space, and proposes a Meta Agent Search algorithm that discovers LLM agents outperforming state-of-the-art human designs.

“More importantly, ADAS introduces the idea of searching the entire agentic system defined in code space, and propose a Meta Agent Search algorithm that discovers LLM agents outperforming state-of-the-art human designs (Hu et al., 2024).”

• cites Hu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The design space is highly extensible, allowing new agentic systems to be integrated as plugin modules.

“Our design space is also highly extensible, allowing new agentic systems to be integrated as plugin modules.”

Classified as probably not a substantial claim.

A more recent work introduces the idea to leverage LLMs to search the entire agentic systems defined in code space, enabling the discovery of agents with more flexible prompts, control flows, etc.

“A more recent work introduces the idea to leverage LLMs to search the entire agentic systems defined in code space (Hu et al., 2024), enabling the discovery of agents with more flexible prompts, control flows, etc.”

• cites Hu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

MoLAS provides standardized IO interfaces for agent modules, facilitating easy recombination of modules from different agentic systems and hence enabling efficient search for novel agents.

“However, MoLAS has a nice feature of providing standardized IO interfaces for agent modules, facilitating easy recombination of modules from different agentic systems and hence enabling efficient search for novel agents.”

Probably not a substantial claim: may not pass S3 (load-bearing).

More experiment results of predicting the performance of the dynamically searched agents are provided in Figure A.15 of the Appendix.

“We provide more experiment results of predicting the performance of the dynamically searched agents in Figure A.15 of the Appendix.”

Classified as probably not a substantial claim.

To ensure a fair comparison, the same number of few-shot examples is used across all methods.

“To ensure a fair comparison, we use the same number of few-shot examples across all methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OPRO and Promptbreeder can be considered as using LLMs to optimize LLM agents defined on prompt space.

“OPRO (Yang et al., 2024) and Promptbreeder (Fernando et al., 2024) can be considered as using LLMs to optimize LLM agent defined on prompt space.”

• cites Yang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fernando et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Although ADAS can in principle discover more sophisticated agents by searching the entire code space, it may require more iterations, and thus higher LLM token usage, than AgentSquare to do so.

“While in principle ADAS has the potential to discover more sophisticated agents by searching in the entire code space, it may require more iterations (and thus higher LLM token usage) to achieve this.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

LLM-based autonomous agents are an advanced AI system that uses a core LLM to manage external functional modules and interact with the world.

“LLM-based autonomous agents are an advanced AI system using a core LLM to manage external functional modules and interact with the world (Ding et al., 2024b).”

• cites Ding et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The tested agents were generated through random sampling by randomly combining existing modules.

“The tested agents were generated through random sampling by randomly combining existing modules.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The lack of a unified design space and consistent terminologies across individual works makes it hard to compare different agents, understand their evolution routes, and guide new agent design directions.

“The lack of a unified design space and consistent terminologies across individual works makes it hard to compare different agents, understand their evolution routes, and guide new agent design directions.”

May combine two claims.

Module recombination significantly expands the search space, reducing the risk of falling into a local optima.

“Module recombination significantly expands the search space, reducing the risk of falling into a local optima.”

May combine two claims.

Engineering-oriented open resources such as LangChain and BabyAGI provide convenient ways to build an LLM-centric agentic system but still need human participation to organize different modules and cannot support optimization of the designed agent.

“Engineering-oriented open resources like LangChain * and BabyAGI † have provided convenient ways to build an LLM-centric agentic system, however, they still need human participation to organize different modules and can't support the optimization of the designed agent.”

• cites LangChainNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites BabyAGINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ADAS searches the entire agentic system defined on code space, enabling the search for LLM agents with more flexible prompts, tool uses and control flows.

“More relevantly, ADAS (Hu et al., 2024) proposes to search the entire agentic system defined on code space, enabling the search for LLM agents with more flexible prompts, tool uses, control flows and more.”

• cites Hu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Both module-level and prompt-level search methods encounter performance bottlenecks during the search process, resulting in sub-optimal agent architectures.

“As a result, they all encounter performance bottlenecks during the search process, resulting in sub-optimal agent architectures.”

Classified as probably not a substantial claim.

The comparison of search-based methods is conducted with a fixed LLM token budget by maintaining the same number of search iterations.

“The comparison of search-based methods is conducted with a fixed LLM token budget to ensure fairness by maintaining the same number of search iterations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The initial agent is set as a random module combination, and the search process terminates after 5 consecutive iterations without performance improvement.

“The initial agent is set as a random module combination, and the search process terminates after 5 consecutive iterations without performance improvement.”

May combine two claims.

LLM-based agent systems have not yet formed a unified design space and an automatic design approach.

“LLM-based agent system, as the most advanced AI system, has not yet formed a unified design space and an automatic design approach.”

Classified as probably not a substantial claim.

The same number of few-shot examples is used across all compared methods to ensure a fair comparison.

“To ensure a fair comparison, we use the same number of few-shot examples across all methods.”