CoChat
Sign Up
Preprint

MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems

Ye, Rui, Tang, Shuo, Ge, Rui, Du, Yaxin, Yin, Zhenfei, Chen, Siheng, Shao, Jing·2025·arXiv
DOI 10.48550/arxiv.2503.03686arXiv 2503.03686
Open this paper in CoChat — free account

Abstract

LLM-based multi-agent systems (MAS) have shown significant potential in tackling diverse tasks. However, to design effective MAS, existing approaches heavily rely on manual configurations or multiple calls of advanced LLMs, resulting in inadaptability and high inference costs. In this paper, we simplify the process of building an MAS by reframing it as a generative language task, where the input is a user query and the output is a corresponding MAS. To address this novel task, we unify the representation of MAS as executable code and propose a consistency-oriented data construction pipeline to create a high-quality dataset comprising coherent and consistent query-MAS pairs. Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within a single LLM inference. The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses. Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consistently outperforms 10+ baseline MAS methods on diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability. Code will be available at https://github.com/rui-ye/MAS-GPT.

AI snapshot

Written from the paper’s full text.

TL;DR

MAS-GPT reframes building an LLM-based multi-agent system (MAS) as a generative language task: given a user query, it generates an executable MAS in one LLM inference. Trained on a consistency-oriented query-MAS dataset, it outperforms 10+ baselines across 9 benchmarks and 5 LLMs.

Key findings

MAS-GPT was trained on 11,442 query-MAS pairs with 7,580 unique MAS designs and, using Llama-3-70B-Instruct to drive the MAS, achieved 65.47 average across 8 benchmarks versus 61.58 for the second-best baseline, a 3.89% improvement. On AIME2024, it improved over a single LLM by 13.34% with o1-preview; the abstract/intro also report 13.3% and 10.0% gains for o1 and DeepSeek-R1 respectively. Ablations show inter-consistency-oriented pair selection contributes an absolute 8.39% gain on MATH (60.26 to 68.65), and both intra-consistency refinement strategies also improve performance. MAS-GPT outperformed math-specific AFlow on MATH by 3.53% and beat ChatDev on HumanEval (91.18 vs 83.33), HumanEval+ (87.23 vs 84.04), and MATH (77.59 vs 62.07).

Methods

The paper represents an executable MAS as Python code: a forward function where agent prompts are variables, LLM calls are functions, and inter-agent interactions are string concatenations. Data construction uses query pools from MATH, GSM8K, MBPP, MMLU, and SciQ; a base MAS pool of over 40 designs; N×M query-MAS evaluation; inter-consistency-oriented selection that clusters similar queries and assigns the highest cumulative-scoring MAS; and intra-consistency-oriented refinement that adjusts agents and adds a reasoning statement, accepting refined MAS only if not worse. MAS-GPT is supervised fine-tuned on Qwen2.5-Coder-32B-Instruct using 16 A100s, effective batch size 32, 3 epochs, and learning rate 1e-5; evaluation covers 9 benchmarks, 5 LLMs, and 10+ baselines.

Limitations

The paper does not include a dedicated limitations section; it only states that potential negative impacts are similar to those of LLMs generally. Evident from the text: training queries are sampled from MATH, GSM8K, MBPP, MMLU, and SciQ, and GPQA and SciBench are explicitly out-of-domain, so generalization beyond these domains is not fully established. Evaluation relies on LLM-based answer extraction and comparison, which may introduce extraction or evaluation errors, and the cost comparison interprets MAS-GPT inference as 0.5 calls based on its 32B versus 70B model size, an assumption rather than a measured cost.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.03686

Cites

Statistics

Nothing to recompute.

Claims the paper makes

4 of 21 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 3

    The paper's unified representation of MAS is motivated by the observation that all existing LLM-based MAS methods are ultimately implemented as code.

    “This unified representation is motivated by the observation that all existing LLM-based MAS methods are ultimately implemented as code, encompassing the definition of agents' prompts, LLM calls, and inter-agent interactions (Qian et al., 2024a; Hu et al., 2024; Zhang et al., 2024).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Qian et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zhang et al., 2024 source

      In the Agentic Workflow section (page unavailable), AFlow adopts code to represent workflow relationships because it can express sequences, conditions, loops, and graph structures. Its Additional information section (page unavailable) treats some multi-agent systems as functionally equivalent to agentic workflows. Neither passage says that all existing LLM-based MAS methods are ultimately implemented as code, or gives that universal observation as the reason for the representation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT outperforms math-specific AFlow on the MATH dataset by 3.53%.

    “As a general method, our MAS-GPT even outperforms math-specific AFlow on the MATH dataset by 3 . 53% !”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhang et al., 2024 source

      Table 1 reports AFlow’s 56.2% solve rate on its MATH test set, and the Conclusion describes a workflow tailored to the MATH task. The paper does not evaluate MAS-GPT or establish that it outperforms AFlow by 3.53%; that comparison is not supported by this source. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingown data
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 4

    The base MAS pool comprises over 40 unique MAS designs.

    “resulting in a base MAS pool comprising over 40 unique MAS designs (Figure 7).”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Abstract · p. 1

    MAS-GPT consistently outperforms 10+ baseline MAS methods on 9 benchmarks and 5 LLMs.

    “Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consistently outperforms 10+ baseline MAS methods on diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability.”
    • Also stated in: § 1. Introduction · p. 2 · hedged
      “Our results show that MAS-GPT consistently outperforms baseline methods on average, indicating its high generality and effectiveness.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT significantly enhances the reasoning performance over single LLM when empowered with o1-preview.

    “(a) Different methods empowered with strong reasoning LLM: o1-preview. We see that our MAS-GPT significantly enhance the reasoning performance over single LLM, indicating its potential in further augmenting LLM reasoning.”
    Quote in the paperSubstantial claim
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 18
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 1

    GPTSwarm and DyLAN rely on LLMs to iteratively adjust collaboration structure or agents' prompts for each task, requiring multiple LLM inferences.

    “For example, both GPTSwarm (Zhuge et al., 2024) and DyLAN (Liu et al., 2024) rely on LLMs to replace human involvement, iteratively adjusting the collaboration structure or agents' prompts in the MAS for each specific task. However, this process often requires multiple LLM inferences.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Zhuge et al., 2024 source
    • cites Liu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisenone given
    § 4.2. Main Results · p. 7

    AFlow needs to call APIs of powerful proprietary LLMs, such as Claude-3.5Sonnet, 10 times per query and depends on a hold-out validation set.

    “while AFlow needs to call the APIs of powerful proprietary LLMs, such as Claude-3.5Sonnet (Anthropic, 2024), 10 times per query and depends on a hold-out validation set.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Anthropic, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zhang et al., 2024 source

      The Datasets and Implementation Details sections (pages not provided) say AFlow uses Claude-3.5-sonnet as an optimizer, accesses its models through APIs, and splits data into validation and test sets. The Expansion section says each generated workflow is tested five times on the validation set; it does not establish a requirement of ten proprietary-LLM API calls per query. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.1. Experimental Setups · p. 5

    The number of training samples (query-MAS pairs) is approximately 11k.

    “The number of training samples (i.e., query-MAS pairs) is approximately 11k; see the statistics of our dataset in Table 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 6

    MAS-GPT significantly outperforms the baseline methods on average, outperforming the second-best method by 3.89%.

    “our MASGPT significantly outperforms the baseline methods on average, outperforming the second-best method by 3 . 89% .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT even outperforms AFlow on its in-domain benchmarks; while AFlow fails on out-of-domain benchmarks.

    “(b) Comparisons with AFlow (optimized on MATH). MAS-GPT even outperforms AFlow on its in-domain benchmarks; while AFlow fails on out-of-domain benchmarks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT achieves the best performance with low inference cost.

    “(c) MAS-GPT achieves the best performance with low inference cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 1. Introduction · p. 2

    MAS-GPT brings 13.3% and 10.0% gain on AIME2024 when combined with o1 and DeepSeek-R1, respectively.

    “Meanwhile, MAS-GPT has the potential to further push the boundary of strong reasoning capability of o1 (OpenAI, 2024b) and DeepSeek-R1 (Guo et al., 2025), bringing 13 . 3% and 10 . 0% gain on AIME2024 (a challenging mathematical benchmark), respectively.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites OpenAI, 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Guo et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 1. Introduction · p. 2

    MAS-GPT can generalize to unfamiliar queries and generate novel MAS via case studies.

    “We also verify that our MAS-GPT can generalize to unfamiliar queries and generate novel MAS via case studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5. Conclusion · p. 8

    MAS-GPT outperforms all baselines, validating its effectiveness and generalizability.

    “The results consistently demonstrate that MAS-GPT outperforms all baselines, strongly validating its effectiveness and generalizability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Analysis of MAS-GPT · p. 8

    Adjusting MAS to make it query-dependent and introducing a reasoning process both contribute to performance improvement.

    “From Table 4, by comparing ➁ and ➃ , ➂ and ➃ , we see that our designs in adjusting MAS and introducing reasoning process both contribute to performance improvement, indicating the effectiveness of our proposed intra-consistency-oriented pair refinement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20

    On HumanEval-Plus, MAS-GPT achieves a score of 87.23, higher than ChatDev's 84.04.

    “| METHOD | HUMANEVAL | HUMANEVAL-PLUS | MATH |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| CHATDEV (QIAN ET AL., 2024A) | 83.33 | 84.04 | 62.07 |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| MAS-GPT (OURS) | 91.18 | 87.23 | 77.59 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Analysis of MAS-GPT · p. 8

    Extractability and executability generally improve with data scale, except for extractability under 0 data sample.

    “Results in Figure 5(a) show that except for the extractability under 0 data sample (the base model knows that it needs to generate Python code, but do not know what codes are needed), the extractability and executability generally improves with the data scale.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 7

    Among the four methods compared, MAS-GPT achieves the best performance while requiring the fewest inference calls.

    “From the figure, we observe that, among the four methods compared, MAS-GPT achieves the best performance while requiring the fewest inference calls, demonstrating its efficiency and effectiveness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT significantly outperforms baseline methods on the AIME2024 mathematical benchmark, improving over the single LLM by 13.34%.

    “The results, as shown in Figure 4(a), show that our proposed MAS-GPT significantly outperforms the baseline methods on this challenging task. Specifically, it can improve over the single LLM by a large margin: 13 . 34 % .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § A.2. MAS-GPT Can Generalize to Unseen Queries · p. 13

    MAS-GPT can generalize to unseen queries and generate appropriate MAS.

    “This indicates that MAS-GPT can generalize to unseen queries and generate appropriate MAS.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20

    On HumanEval, MAS-GPT achieves a score of 91.18, higher than ChatDev's 83.33.

    “| METHOD | HUMANEVAL | HUMANEVAL-PLUS | MATH |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| CHATDEV (QIAN ET AL., 2024A) | 83.33 | 84.04 | 62.07 |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| MAS-GPT (OURS) | 91.18 | 87.23 | 77.59 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 18

    MAS-GPT analyzes the domain, difficulty, and required knowledge.

    “MAS-GPT firstly analyzes the domain, difficulty, and required knowledge”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20

    On MATH, MAS-GPT achieves a score of 77.59, higher than ChatDev's 62.07.

    “| METHOD | HUMANEVAL | HUMANEVAL-PLUS | MATH |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| CHATDEV (QIAN ET AL., 2024A) | 83.33 | 84.04 | 62.07 |”
    • Also stated in: § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20 · plain
      “| MAS-GPT (OURS) | 91.18 | 87.23 | 77.59 |”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4.2. Main Results · p. 7

    MAS-GPT consistently performs the best across the evaluated benchmarks (MATH, GSM8K, MMLU, HumanEval+).

    “In contrast, our MAS-GPT consistently performs the best across these benchmarks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 18

    MAS-GPT proposes the idea of what multi-agent system is appropriate.

    “then propose the idea of what multi-agent system is appropriate”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Study design and conductnone given
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 5

    The refined MAS is tested and accepted only if it achieves a not-worse score than the base MAS.

    “the refined MAS refine i is tested and accepted only if it achieves a not-worse score.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4.1. Experimental Setups · p. 6

    The paper considers 10 baselines that are suitable for handling diverse tasks.

    “For fair comparisons, we consider 10 baselines that are suitable for handling diverse tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 5

    The refined dataset ensures both inter- and intra-consistency, providing high-quality training data.

    “This refined dataset ensures both inter- and intra-consistency, providing high-quality training data for subsequent model fine-tuning.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 1. Introduction · p. 2

    The paper unifies the representation of MAS by describing it as a Python code snippet (a forward function), with each agent's prompt as a variable, LLM calls as functions, and agent relationships as string concatenation.

    “To ensure the generated MAS is readily executable, we unify the representation of MAS by describing it as a Python code snippet (i.e., a forward function), with each agent's prompt as a variable, LLM calls as functions, and agent relationships as string concatenation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    MAS-GPT has high effectiveness, efficiency and strong generalization ability.

    “indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning
    § 1. Introduction · p. 2

    The inter-consistency-oriented selection approach ensures that similar queries are paired with similar high-performing MAS.

    “During selection, we design an inter-consistency-oriented selection approach to ensure that similar queries are paired with similar high-performing MAS, facilitating the model to learn generalizable patterns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 3.3. Discussions · p. 5

    Training incurs some cost but is a one-time expense, whereas inference is potentially endless in practical applications.

    “Although training incurs some cost, it is a one-time expense, whereas inference is potentially endless in practical applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.1. Overall System Integrated with MAS-GPT · p. 3

    The paper's approach minimizes time and computational resources traditionally required to build the right MAS and enhances flexibility.

    “This approach not only minimizes the time and computational resources traditionally required to build the right MAS but also enhances the system's flexibility by generating taskspecific solutions in real-time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    MAS-GPT is an open-source medium-sized LLM capable of generating query-adaptive MAS within a single LLM inference.

    “Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within a single LLMinference.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Abstract · p. 1

    The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses.

    “The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning
    § 1. Introduction · p. 2

    The intra-consistency-oriented refinement method strengthens the relatedness between query and MAS.

    “During refinement, we propose a intra-consistency-oriented refinement method to strengthen the relatedness between query and MAS, enabling the model to learn the reasonable correlation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.3. Discussions · p. 5

    The paper's system offers key advantages: simplicity, cost-efficiency, and adaptability (generality).

    “our system integrated with MAS-GPT offers the following key advantages: simplicity, cost-efficiency, and adaptability (generality).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 3.3. Discussions · p. 5

    MAS-GPT has the potential to further advance the real-world application of MAS due to its simplicity, cost-efficiency, and adaptability.

    “We believe that MAS-GPT has the potential to further advance the real-world application of MAS due to its simplicity, cost-efficiency, and adaptability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 1

    MAS in MetaGPT, ChatDev, and AgentVerse are manually crafted, with predetermined and static collaboration structure and agents' prompts, lacking generality to adapt to any given tasks.

    “MAS in MetaGPT (Hong et al., 2024), ChatDev (Qian et al., 2024a), and AgentVerse (Chen et al., 2024) are all manually crafted (e.g., for coding tasks). That is, the collaboration structure and agents' prompts are predetermined and static, lacking in the generality to adapt towards any given tasks.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Hong et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Qian et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationown data
    § 5. Conclusion · p. 8

    MAS-GPT has the potential to further enhance state-of-the-art reasoning capabilities.

    “Additionally, we observe MAS-GPT's potential to further enhance state-of-the-art reasoning capabilities, as well as its scalability for continued improvements.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductreasoning
    § C.2. Evaluations · p. 20

    The evaluation uses a two-step process based on AutoGen prompts: first extract the answer using LLMs, then compare with ground truth, to avoid extraction errors.

    “To address this issue, we adopt a two-step evaluation process based on the prompts used in AutoGen (Wu et al., 2023), first extract the answer, then evaluation.”
    • Also stated in: § C.2. Evaluations · p. 20 · plain
      “To avoid extraction errors that could impact the evaluation of MAS performance, we use LLMs to automate the answer extraction process.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wu et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 3.1. Overall System Integrated with MAS-GPT · p. 3

    MAS-GPT dynamically creates an MAS for each query, ensuring that the system adapts to a wide range of tasks.

    “Instead of relying on pre-built agent configurations, MASGPT dynamically creates an MAS for each query, ensuring that the system adapts to a wide range of tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 5

    The paper's intra-consistency-oriented refinement enables the model to better understand the context and logic behind each decision, which in turn facilitates model training and improves generalization.

    “This process enables the model to better understand the context and logic behind each decision, which in turn facilitates model training and improves generalization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems · p. 20

    ChatDev is specifically designed for software development.

    “Table 8: Comparisons with task-specific method, ChatDev (Qian et al., 2024a), which is specifically designed for software development.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Qian et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § C.2. Evaluations · p. 20

    Responses generated by multi-agent systems are often unstructured and irregular, making it difficult to extract the final answer using rule-based methods.

    “Here the responses generated by multi-agent systems (MAS) are often unstructured and irregular, making it difficult to extract the final answer to a query using rule-based methods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2. MAS-GPT: Dataset Construction and Training · p. 4

    The paper's inter-consistency-oriented pair selection improves inter-consistency of query-MAS pairs, facilitating the model in learning generalizable patterns and generalize across similar queries.

    “By aligning similar queries with the same high-performing MAS, this method improves the inter-consistency of the query-MAS pairs, facilitating the model in learning to recognize generalizable patterns and generalize across similar queries.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 5. Conclusion · p. 8

    MAS-GPT is scalable for continued improvements.

    “Additionally, we observe MAS-GPT's potential to further enhance state-of-the-art reasoning capabilities, as well as its scalability for continued improvements.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § C.2. Evaluations · p. 21

    The evaluation of MAS performance on coding tasks uses a two-step approach: first prompting the LLM to extract the code from the MAS response, and second executing it in a coding environment to calculate the pass rate.

    “with a two step approach: first, prompting the LLM to extract the code from the MAS response, and second, executing it in a coding environment to calculate the pass rate”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § C.2. Evaluations · p. 21

    The MAS performance on coding tasks is evaluated based on pass rate on test cases.

    “We evaluate the MAS performance on coding tasks based on pass rate on test cases”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data
    § 4.2. Main Results · p. 7

    MAS-GPT has promising potential in pushing the boundaries of LLM reasoning.

    “This result not only verifies the generality of our proposed MAS-GPT, but also indicates its promising potential in pushing the boundaries of LLM reasoning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § A.3. MAS-GPT Can Generate Novel MAS · p. 13

    MAS-GPT is not merely memorizing the data during training, but actually learns patterns of designing MAS, indicating generalization ability.

    “This shows that MAS-GPT is not merely memorizing the data during training, but actually learns patterns of designing MAS, indicating the generalization ability of our proposed MAS-GPT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § C.2. Evaluations · p. 20

    For code benchmarks like HumanEval and MBPP, correctness is assessed using test cases.

    “For code benchmarks like HumanEval and MBPP, we assess correctness using test cases.”
    Quote in the paperNot a substantial claim
47 of 47

  • The paper relies on this without support.

    LLMs have limited (if any) knowledge of MAS generation.

    “Unlike realworld dialogue data, LLMs have limited (if any) knowledge of MAS generation.”

  • The paper relies on this without support.

    Advanced LLMs could generate inappropriate or non-executable MAS.

    “Next, we infer and evaluate the refined MAS on the corresponding query, as advanced LLMs could generate inappropriate or non-executable MAS.”

  • The paper relies on this without support.

    The alignment between a query and its associated MAS may still be suboptimal.

    “Specifically, the alignment between a query and its associated MAS may still be suboptimal, making it challenging for the model to learn meaningful associations.”
3 of 3

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Research works printed but not linked: 6

  1. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning , 2024.
  2. Computational Linguistics (Volume 1: Long Papers) , pp. 15174-15186, 2024a.
  3. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problemsolving abilities of large language models. In Forty-first International Conference on Machine Learning , 2024a.
  4. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Selfconsistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2024b.
  5. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations , 2024.
  6. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y ., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems , 36, 2023.

Authors
Ye, R., Tang, S., Ge, R., Du, Y., Yin, Z., Chen, S., & Shao, J.
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2503.03686
arXiv
2503.03686
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review