CoChat
Sign Up
PreprintCited by 45Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Mixture-of-Agents Enhances Large Language Model Capabilities

Open access
Wang, Junlin, Wang, Jue, Athiwaratkun, Ben, Zhang, Ce, Zou, James·2024·arXiv
DOI 10.48550/arxiv.2406.04692arXiv 2406.04692
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes Mixture-of-Agents (MoA), a layered, prompt-only framework where multiple LLMs iteratively refine responses by using outputs from agents in the previous layer. MoA achieves state-of-the-art results on AlpacaEval 2.0, MT-Bench, and FLASK, with an open-source-only MoA reaching 65.1% LC win rate on AlpacaEval 2.0 versus 57.5% for GPT-4 Omni.

Key findings

  • On AlpacaEval 2.0, MoA with only open-source models scored 65.1±0.6% LC win rate, a 7.6% absolute improvement over GPT-4 Omni’s 57.5%; MoA w/ GPT-4o scored 65.7±0.7%, and MoA-Lite scored 59.3±0.2%.
  • On MT-Bench, MoA w/ GPT-4o scored 9.40±0.06, MoA scored 9.25±0.10, and MoA-Lite scored 9.18±0.09, securing top leaderboard positions.
  • On FLASK, MoA improved robustness, correctness, efficiency, factuality, commonsense, insightfulness, and completeness over the single aggregator Qwen-110B-Chat, and outperformed GPT-4 Omni in correctness, factuality, insightfulness, completeness, and metacognition; conciseness was marginally worse due to more verbose outputs.
  • MoA outperformed an LLM-ranker baseline; performance increased monotonically with the number of proposers, and multiple-proposer setups beat single-proposer setups (e.g., n=6: 61.3% vs. 56.7%). MoA-Lite matched GPT-4o’s cost while achieving higher quality and outperformed GPT-4 Turbo by approximately 4% while being more than twice as cost-effective.

Methods

  • MoA uses a layered architecture with n LLM agents per layer; each agent receives all outputs from the previous layer as auxiliary information and generates a refined response. The method requires no fine-tuning and operates only through prompting and generation.
  • The default MoA used six open-source models (Qwen1.5-110B-Chat, Qwen1.5-72B-Chat, WizardLM-8x22B, LLaMA-3-70B-Instruct, Mixtral-8x22B-v0.1, dbrx-instruct), three MoA layers, the same model set in each layer, and Qwen1.5-110B-Chat as the final aggregator. MoA-Lite used two layers and Qwen1.5-72B-Chat as aggregator; MoA w/ GPT-4o used GPT-4o as the final aggregator.
  • Evaluation used AlpacaEval 2.0, MT-Bench, and FLASK, plus a MATH task and budget/token analyses. AlpacaEval used LC win rate; MT-Bench used GPT-4 grading; FLASK provided 12 skill-specific scores. AlpacaEval experiments were run three times and averaged with standard deviation.

Limitations

  • The paper states that iterative aggregation means the model cannot decide the first token until the last MoA layer is reached, potentially causing high Time to First Token (TTFT) and negatively impacting user experience; it suggests limiting MoA layers or exploring chunk-wise aggregation.
  • The text makes evident that MoA requires multiple LLM calls and iterative aggregation, increasing inference cost and latency; the paper analyzes cost and tflops but does not eliminate this overhead.
  • The evaluation relies heavily on LLM-based judges (GPT-4-based evaluator, AlpacaEval LC, MT-Bench GPT-4 grading), which may inherit judge biases; the paper does not report a separate human evaluation beyond citing AlpacaEval’s correlation with human preferences.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.04692

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 7 cited sources checked · 4 supported · 0 flagged

  • Findingown data

    The Mixture-of-Agents approach is applicable to reasoning tasks, such as those in the MATH dataset.

    “Here, we demonstrate that our approach is applicable to reasoning tasks, such as those in the MATH dataset Hendrycks et al. (2021).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Hendrycks et al. (2021) source
    • Rests on an assumption: The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.
    • Rests on an assumption: The MATH dataset is representative of reasoning tasks in general.
  • Findingown data

    If prioritizing quality, MoA is the best configuration.

    “Specifically, if we prioritize the quality, MoA is the best configuration.”
    Quote in the paperSubstantial claim
  • Findingown data

    Using multiple different LLMs as proposers consistently yielded better results than using a single LLM as proposer.

    “Overall, using multiple different LLMs consistently yielded better results.”
    Quote in the paperSubstantial claim
  • Findingown data

    A Pareto front can be observed in the relationship between LC win rate and number of tflops.

    “Similar to the cost efficiency analysis, a Pareto front can be observed here as well.”
    Quote in the paperNot a substantial claim
  • Findingown data

    An LLM tends to generate better responses when presented with outputs from other models, even if these other models are less capable by themselves.

    “We identify an inherent phenomenon we term the collaborativeness of LLMs -wherein an LLM tends to generate better responses when presented with outputs from other models, even if these other models are less capable by itself.”
    • Also stated in: § 1 Introduction · boosted
      “Remarkably, this improvement occurs even when the auxiliary responses provided by the other models are of lower quality than what an individual LLM could generate independently.”
    Quote in the paperSubstantial claim
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 3
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Different LLMs possess unique strengths and specialize in various tasks aspects.

    “At the same time, different LLMs possess unique strengths and specialize in various tasks aspects.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Xu et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Roziere et al., 2023 source
    • cites Guo et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The LC metric tracks closely with human preferences, achieving a Spearman correlation of 0.98 with actual human evaluations.

    “This metric tracks closely with human preferences, achieving a Spearman correlation of 0.98 with actual human evaluations(Dubois et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dubois et al., 2024 source

  • May combine two claims.

    Findingown data

    MoA-Lite outperforms GPT-4 Turbo by approximately 4% while being more than twice as cost-effective.

    “Notably, it outperforms GPT-4 Turbo by approximately 4% while being more than twice as cost-effective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Mixture-of-Agents approach lies on the Pareto frontier of performance versus cost, while GPT-4 Turbo and GPT-4o are not cost optimal and are more expensive compared to MoA approaches of the same LC win rate.

    “We show that the Mixture-of-Agents approach lie on this Pareto front, as opposed to GPT-4 Turbo and GPT-4o which are not cost optimal and is more expensive compared to MoA approaches of the same LC win rate.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.

  • May combine two claims.

    Findingown data

    The aggregated response generated by Qwen1.5-110B-Chat reflects a high preference for its own content but also incorporates key points from Llama-3-70B-Instruct and WizardLM 8x22B.

    “The aggregated response generated by Qwen1.5-110B-Chatreflects a high preference for its own content but also incorporates key points from Llama-3-70B-Instruct and WizardLM 8x22B.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    WizardLM demonstrated excellent performance as a proposer model but struggled to maintain its effectiveness in aggregating responses from other models.

    “In contrast, WizardLM demonstrated excellent performance as an proposer model but struggled to maintain its effectiveness in aggregating responses from other models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MoA outperforms GPT-4 Omni in correctness, factuality, insightfulness, completeness, and metacognition.

    “Additionally, MoA also outperforms GPT-4 Omni in terms of correctness, factuality, insightfulness, completeness, and metacognition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MoA did not do as well as GPT-4 Omni on conciseness; its outputs were marginally more verbose.

    “One metric where MoA did not do as well was conciseness; the model produced outputs that were marginally more verbose.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MoA-Lite can match GPT-4o's cost while achieving a higher level of quality.

    “if we want to strike a good balance between quality and cost, MoA-Lite can match GPT-4o's cost while achieving higher level of quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is a positive correlation between win rate and both TF-IDF similarity and Levenshtein similarity.

    “As shown in Figure 6, there is indeed a positive correlation between win rate and both TF-IDF similarity and Levenshtein similarity.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.

  • May combine two claims.

    Findingown data

    MoA achieves top positions on the AlpacaEval 2.0 leaderboard, demonstrating an 8.2% absolute improvement over the previous top model GPT-4o.

    “our MoA methodology achieved top positions on the AlpacaEval 2.0 leaderboard, demonstrating a remarkable 8.2% absolute improvement over the previous top model, GPT-4o.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4's preference score for WizardLM 8x22B's response is 0.99, and the final aggregated answer also achieves a preference score of 0.99.

    “Notably, GPT-4's preference score for WizardLM 8x22B's response is 0.99, and the final aggregated answer also achieves a preference score of 0.99.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MoA excels in robustness, correctness, efficiency, factuality, commonsense, insightfulness, and completeness compared to the single model score of the aggregator Qwen-110B-Chat.

    “our methodology shows significant improvement in robustness, correctness, efficiency, factuality, commonsense, insightfulness, completeness, compared to the single model score of the aggregator, Qwen-110B-Chat.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Empirical evaluations on AlpacaEval 2.0, MT-Bench, and FLASK demonstrated substantial improvements in response quality, with the approach achieving an LC win rate up to 65%.

    “Empirical evaluations conducted on AlpacaEval 2.0, MT-Bench, and FLASK demonstrated substantial improvements in response quality, with our approach achieving the LC win rate up to 65%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.

  • May combine two claims.

    Findingown data

    MoA achieves state-of-the-art performance on AlpacaEval 2.0, MT-Bench, and FLASK, surpassing GPT-4 Omni.

    “MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni.”
    • Also stated in: § 1 Introduction · plain
      “our MoA framework achieves state-of-the-art performance on these benchmarks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Several implementations of MoA can deliver performance comparable to GPT-4 Turbo while being 2× more cost-effective.

    “several implementations of MoA can deliver performance comparable to GPT-4 Turbo while being 2× more cost-effective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MoA using only open-source LLMs achieves a score of 65.1% on AlpacaEval 2.0, compared to 57.5% by GPT-4 Omni.

    “our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.”
    • Also stated in: § 3.2 Benchmark Results · plain
      “achieving a margin of 7.6% absolute improvement from 57.5% (GPT-4o) to 65.1% (MoA).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    There is a positive correlation between the aggregator's response similarity to proposers' responses (BLEU) and the preference scores determined by the GPT-4 based evaluator.

    “The results in Figure 4 indeed confirms a positive correlation between the win rate and the BLEU score.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    The first response aggregation has the most significant boost on generation quality.

    “as the first response aggregation has the most significant boost on generation quality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MoA-Lite achieves 59.3% on AlpacaEval 2.0, a 1.8% improvement over GPT-4o's 57.5%.

    “Even with this lighter approach, we still outperform the best model by 1.8%, improving from 57.5% (GPT-4o) to 59.3% (MoA-Lite).”
    • Also stated in: § 3 Evaluation · plain
      “This makes it more cost-effective than GPT-4o while achieving a 1.8% improvement in quality on AlpacaEval 2.0.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Mixture-of-Agents method harnesses the collective strengths of agents in the Mixture-of-Agents family and can significantly improve upon the output quality of each individual model.

    “Our method harnesses the collective strengths of agents in the Mixture-of-Agents family, and can significantly improve upon the output quality of each individual model.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.

  • May combine two claims.

    Interpretationreasoning

    The proposed method requires iterative aggregation of model responses, so the model cannot decide the first token until the last MoA layer is reached.

    “Our proposed method requires iterative aggregation of model responses, which means the model cannot decide the first token until the last MoA layer is reached.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The requirement to wait until the last MoA layer potentially results in high Time to First Token (TTFT), which can negatively impact user experience.

    “This potentially results in a high Time to First Token (TTFT), which can negatively impact user experience.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The experiments were run three times and the average scores along with the standard deviation were reported.

    “We ran our experiments three times and reported the average scores along with the standard deviation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    MoA provides flexibility and scalability: it can be applied to the latest LLMs regardless of their size or architecture.

    “It provides flexibility and scalability: our method can be applied to the latest LLMs regardless of their size or architecture.”
    Quote in the paperSubstantial claim
25 of 25

  • The paper relies on this without support.

    Scaling up LLMs is exceptionally costly, often requiring extensive retraining on several trillion tokens.

    “Further scaling up these models is exceptionally costly, often requiring extensive retraining on several trillion tokens.”

  • The paper relies on this without support.

    Current models already perform exceptionally well on MT-Bench, as a single model alone can achieve scores greater than 9 out of 10.

    “current models already perform exceptionally well on this benchmark, as a single model alone can achieve scores greater than 9 out of 10.”

  • The paper relies on this without support.

    The benchmarks used (AlpacaEval 2.0, MT-Bench, FLASK, MATH) are valid and representative measures of general LLM capabilities.

    “Empirical evaluations conducted on AlpacaEval 2.0, MT-Bench, and FLASK demonstrated substantial improvements in response quality, with our approach achieving the LC win rate up to 65%.”

  • The paper relies on this without support.

    The MATH dataset is representative of reasoning tasks in general.

    “Here, we demonstrate that our approach is applicable to reasoning tasks, such as those in the MATH dataset Hendrycks et al. (2021).”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The observation that MoA scales further than debate may be explained by its architecture: adding layers does not increase context length per model, while increasing debate rounds expands context and computational overhead.

    “The observation that MoA scales further than debate may be explained by its architecture.”

    From Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling (2026) · cites this paper as Wang et al., 2024a

    Section 2.2 describes MoA’s layered architecture and says it need not concatenate the prompt with all model responses. But the paper does not report that MoA scales further than debate or compare how added layers and debate rounds affect context length and computational overhead; its discussion of debate in Section 4.2 is background only. The proposed explanation is therefore not established by this source. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Existing multi-LLM frameworks rely on fixed mechanisms such as consensus voting or stepwise reasoning to aggregate independent LLM outputs but lack dynamic, debate-style interaction essential for robust reasoning.

    “Prior studies rely on fixed mechanisms such as consensus voting8,9 or stepwise reasoning10 to aggregate independent LLM outputs but lack dynamic, debate-style interaction essential for robust reasoning.”

    From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [10]

    The Introduction describes MoA as aggregating independently generated responses through successive layers of refinement, supporting the stepwise-aggregation part of the claim. But §4.2 Model Ensemble describes prior multi-agent studies in which agents discuss problems interactively, contrary to the blanket assertion that prior frameworks lack debate-style interaction. The paper also does not establish that debate is essential for robust reasoning. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Mixture-of-Agents primarily refine immediate predecessor outputs without revisiting earlier contexts, a limitation that may restrict their applicability in dynamic clinical discussions.

    “yet approaches such as Mixture-of-Agents,10 although introducing hierarchical architectures and network-based communication, primarily refine immediate predecessor outputs without revisiting earlier contexts, a limitation that may restrict their applicability in dynamic clinical discussions.”

    From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [10]

    The Introduction and §2.2 describe successive layers refining responses from the preceding layer, supporting that part of the claim. But §2.2 also says each layer’s input includes the original prompt, so the paper does not support an unqualified claim that MoA never revisits earlier context. It does not assess applicability to dynamic clinical discussions. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • To mitigate model-specific biases, the study adopts the single-proposer setting, where only a single LLM is used for all proposer models.

    “To mitigate model-specific biases, we adopt the singleproposer setting (Wang et al., 2024a), where only a single LLM is used for all proposer models.”

    From Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling (2026) · cites this paper as Wang et al., 2024a

  • Multi-agent systems can outperform single-agent baselines but typically require more compute.

    “Multi-agent systems can outperform single-agent baselines (Wang et al., 2024a;Du et al., 2023), but they typically require more compute.”

    From Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling (2026) · cites this paper as Wang et al., 2024a

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Multiple-proposer scores do not increase monotonically

    Minor

    The multiple-proposer score falls from 58.8% at two proposers to 58.0% at three, contrary to the stated monotonic increase.

  • Headline AlpacaEval score differs from the reported top result

    Minor

    The introduction reports a new SOTA win rate of 65.8%, while Table 2 lists the highest AlpacaEval LC win rate as 65.7%.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J.
Affiliations
  • Together (US)
  • University of Chicago (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2406.04692
arXiv
2406.04692
OpenAlex
W4399511904
Cited by
25
References
43
Citations by year
2026: 4 · 2025: 17 · 2024: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
mixture-of-agents, large language models, multi-agent architecture, agent cooperation, open-source LLMs, MT-Bench, Flask

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review