Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Open accessAbstract
Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves $6.6\%$ improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of $3.8\%$ improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.
AI snapshot
Written from the paper’s full text.
TL;DR
Self-MoA, which ensembles repeated samples from a single top-performing LLM, generally outperforms standard Mixture-of-Agents (Mixed-MoA) that mixes different LLMs, achieving 6.6% improvement on AlpacaEval 2.0 and an average 3.8% improvement across MMLU, CRUX, and MATH. The paper attributes this to a quality-diversity trade-off in which MoA is more sensitive to proposer quality than diversity, and introduces a sequential Self-MoA-Seq for context-limited scaling.
Key findings
- On AlpacaEval 2.0, Self-MoA + WizardLM-2-8x22B reached a 65.7 LC win rate versus 59.1 for 2-Layer MoA, a 6.6 point improvement; with top models gemma-2-9b-it-WPO-HB and gemma-2-9b-it-SimPO, Self-MoA achieved 78.5 and 75.0 LC win rates and secured the top leaderboard position during submission.
- Across MMLU, CRUX, and MATH, Self-MoA with the TaskBest model consistently outperformed all 13 tested Mixed-MoA configurations, with an average 3.8% improvement; in a constructed mixture task, only mmdddd and mmmddd slightly outperformed Self-MoA dddddd, by 0.17% and 0.35%.
- Quality-diversity analysis found MoA performance positively correlated with both quality and diversity, with linear regression p-values < 0.001 and R² around 0.7; α was greater than β across all three datasets, indicating greater sensitivity to quality, and best performance typically occurred in high-quality, relatively low-diversity regions.
- Self-MoA-Seq performed comparably to or slightly better than Self-MoA while handling arbitrary numbers of outputs; adding Llama-3.1-8B-Instruct to 30 Qwen2-7B-Instruct samples boosted accuracy by around 2%, whereas adding DeepSeek-Coder-V2-Lite-Instruct reduced accuracy by more than 1.5%.
Methods
- AlpacaEval 2.0 experiments used six open-source instruction-tuned models (Qwen1.5-110B-Chat, Qwen1.5-72B-Chat, WizardLM-8x22B, LLaMA-3-70B-Instruct, Mixtral-8x22B-Instruct-v0.1, dbrx-instruct) with temperature 0.7 and Qwen1.5-110B-Chat as aggregator; Self-MoA aggregated six outputs from WizardLM-2-8x22B. Top-model experiments used gemma-2-9b-it-WPO-HB and gemma-2-9b-it-SimPO with four outputs due to context constraints.
- Specialized-model experiments used Qwen2-7B-Instruct (i), DeepSeek-Coder-V2-Lite-Instruct (d), and Qwen2-Math-7B-Instruct (m) on MMLU-redux (3,000 samples), CRUX (800 Python functions), and the MATH test subset (5,000 samples), fixing six proposers and sweeping 13 combinations, mostly with Qwen2-7B-Instruct as aggregator.
- Quality-diversity analysis involved over 70 unique MoA proposer mixtures and over 200 experiments; diversity was measured with the Vendi Score, quality as average proposer accuracy, and linear regression fit t = αq + βd + γ. Self-MoA-Seq used a sliding window of size six and varied samples from 6 to 30.
Limitations
- The paper states Self-MoA cannot scale indefinitely by simply increasing computation because repeated-sampling diversity tends to plateau, aggregating many samples is harder for LLMs, and context length limits restrict how many responses an aggregator can process at once (e.g., 8192 tokens for Gemma 2); Self-MoA-Seq addresses the latter two.
- The paper states the MATH linear-regression trend was inconsistent because the aggregator Qwen2-7B-Instruct is relatively weak on MATH compared to the strongest individual model, Qwen2-Math-7B-Instruct, constraining MoA performance.
- Evident from the text: conclusions depend on aggregator and proposer quality; in the mixture task Mixed-MoA can slightly outperform Self-MoA (0.17% and 0.35%), and adding a weaker model reduced accuracy by more than 1.5%, so benefits are setting-dependent.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
5 of 16 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured
- Findingown data
On MMLU, where Qwen2-7B-Instruct is the strongest individual model, the R^2 value can exceed 0.9 with K=2 using the Centered-1/K-Norm.
“on MMLU, where Qwen2-7B-Instruct is the strongest individual model, we find that the R 2 value can exceed 0.9 with K = 2 using the Centered-1/K-Norm.”
Quote in the paperNot a substantial claim - Findingown data
The linear model based on quality and diversity explains approximately 70% of MoA's performance (R^2 around 0.7).
“The R 2 values from the linear regression across three datasets are approximately around 0.7, indicating that the linear model based on quality and diversity explains 70% MoA's performance”
Quote in the paperSubstantial claim - Findingown data
Self-MoA achieves a 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark.
“Self-MoA achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark”
- Also stated in: § 1 Introduction · plain
“Self-MoA on the strongest model achieves 6.6 point improvement over its mixed counterpart on the AlpacaEval 2.0 benchmark”
- Also stated in: § Models. · plain
“outperforming the Mixed-MoA baseline by 6.6 point.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
In a constructed mixture task tailored for Mixed-MoA where each individual model excels in a specific subtask, only two Mixed-MoA strategies slightly outperform Self-MoA by 0.17% and 0.35%.
“even in a constructed mixture task tailored for Mixed-MoA where each individual model excels in a specific subtask, only two Mixed-MoA strategies slightly outperform Self-MoA by 0.17% and 0.35%.”
- Also stated in: § 4.2 When Mixed-MoA Outperforms Self-MoA? · plain
“Mixed-MoA of mmdddd and mmmddd achieves the average performance of 59.86% and 60.04%, improves upon Self-MoA of dddddd by 0.17% and 0.35%.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 When Mixed-MoA Outperforms Self-MoA? · plain
- Findingown data
Self-MoA, which aggregates outputs from only the single top-performing LLM, outperforms standard MoA that mixes different LLMs in a large number of scenarios.
“Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios”
- Also stated in: § 1 Introduction · plain
“Surprisingly, we find that Mixed-MoA is usually sub-optimal compared with Self-MoA, especially when there exist significant quality differences among proposers.”
- Also stated in: § 1 Introduction · boosted
“Surprisingly, it demonstrates superior performance compared to existing Mixed-MoA approaches, which emphasize cross-model diversity, across a wide range of benchmarks.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
Adopting a task-specific aggregator yields an additional performance boost of 1-2 points.
“adopting a task-specific aggregator yields an additional performance boost of 1-2 points.”
Quote in the paperSubstantial claim - Findingown data
Mixing different LLMs often lowers the average quality of the models.
“mixing different LLMs often lowers the average quality of the models.”
Quote in the paperSubstantial claim - Findingown data
Increasing model diversity does not always lead to better performance.
“Interestingly, increasing model diversity does not always lead to better performance.”
Quote in the paperSubstantial claim - Findingown data
MoA performance is rather sensitive to the quality of the proposers.
“We confirm that the MoA performance is rather sensitive to the quality”
- Also stated in: § 1 Introduction · plain
“Interestingly, we find that MoA is quite sensitive to variations in quality, with optimal performance typically occurring in regions characterized by high quality and relatively low diversity.”
- Also stated in: § 4 The Quality-Diversity Trade-off · boosted
“Notably, the best performance of MoA is typically observed in the bottom right of each subplot, indicating a strong sensitivity to quality.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
Self-MoA with the TaskBest model consistently outperforms all 13 tested Mixed-MoA configurations across all tasks (MMLU, CRUX, MATH).
“When using i as the aggregator, Self-MoA with the TaskBest model consistently outperforms all 13 tested Mixed-MoA configurations across all tasks.”
- Also stated in: § 1 Introduction · plain
“employing task-specific models as proposers for Self-MoA can significantly outperform the best Mixed-MoA.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
Self-MoA achieves an average of 3.8% improvement across various benchmarks, including MMLU, CRUX, and MATH.
“an average of 3.8% improvement across various benchmarks, including MMLU, CRUX, and MATH”
- Also stated in: § 4.2 When Mixed-MoA Outperforms Self-MoA? · plain
“Self-MoA with TaskBest models achieve an average of 3.8% improvement from Table 6”
Quote in the paperSubstantial claim - Also stated in: § 4.2 When Mixed-MoA Outperforms Self-MoA? · plain
- Findingown data
Employing Centered-1/K-Norm with K=2 achieves strong performance across all three datasets.
“Overall, we conclude that employing Centered-1/K-Norm with K = 2 (marked in blue) achieves strong performance across all three datasets.”
Quote in the paperSubstantial claim - Findingown data
There exists a trade-off between diversity and quality among the proposers in MoA.
“we uncover a core trade-off between diversity and quality among the proposers”
- Also stated in: § 4 The Quality-Diversity Trade-off · plain
“There exists a trade-off in the achievable Pareto front between diversity and quality.”
Quote in the paperSubstantial claim - Also stated in: § 4 The Quality-Diversity Trade-off · plain
- Findingown data
Self-MoA applied to the two top-performing models on AlpacaEval 2.0 (gemma-2-9b-it-WPO-HB and gemma-2-9b-it-SimPO) achieves a 2-3 point gain over the individual models.
“Self-MoA on two best-performed models on AlpacaEval 2.0 consistently achieves a 2-3 point gain”
- Also stated in: § Model Configuration LC Win Rate · plain
“Self-MoA consistently achieves a 2-3 point gain”
Quote in the paperSubstantial claim - Also stated in: § Model Configuration LC Win Rate · plain
- Findingown data
Self-MoA utilizing six Llama-3.1-8B-Instruct proposers outperforms the Mixed-MoA configuration (iiilll) even when the performance of two individual models is close.
“Table 7 demonstrates that even when the performance of two individual models is close, Self-MoA-utilizing six Llama-3.1-8B-Instructproposers (denoted as llllll)-still outperforms the Mixed-MoA configuration (denoted as iiilll).”
Quote in the paperSubstantial claim - Findingown data
Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves new state-of-the-art performance on the leaderboard.
“Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard.”
- Also stated in: § 1 Introduction · plain
“Self-MoA on two best-performed models on AlpacaEval 2.0 consistently achieves a 2-3 point gain and secures the top position among non-adversarial methods on the leaderboard”
- Also stated in: § Model Configuration LC Win Rate · plain
“Self-MoA consistently achieves a 2-3 point gain and secures the top position on the leaderboard during submission.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingreasoning
This trend is inconsistent for MATH.
“However, this trend is inconsistent for MATH.”
Quote in the paperSubstantial claim - Findingown data
Adding more samples can have both positive and negative effects on performance.
“Adding more samples can have both positive and negative effects, meaning there is no universal compute-optimal solution.”
Quote in the paperSubstantial claim - Findingown data
Self-MoA outperforms its Mixed-MoA counterpart on MT-Bench.
“Table 10 shows that Self-MoA outperforms its Mixed-MoA counterpart, and using GPT-4o as the aggregator can achieve the best performance even with fewer forward passes compared to 3-Layer MoA with GPT-4o.”
Quote in the paperSubstantial claim - Findingown data
Adding DeepSeek-Coder-V2-Lite-Instruct reduces accuracy by more than 1.5% in Self-MoA-Seq after aggregating 30 samples from Qwen2-7B-Instruct.
“adding DeepSeek-Coder-V2-Lite-Instruct reduces accuracy by more than 1.5%”
Quote in the paperSubstantial claim - Findingown data
DeepSeek-Coder-V2-Lite-Instruct underperforms Qwen2-7B-Instruct by 5% on MMLU.
“DeepSeek-Coder-V2-Lite-Instruct (which underperforms Qwen2-7B-Instruct by 5%)”
Quote in the paperSubstantial claim - Findingown data
Mixed-MoA indeed outperforms Self-MoA of dddddd in the constructed mixture task.
“From Table 6, we observe that Mixed-MoA indeed outperforms Self-MoA of dddddd.”
Quote in the paperNot a substantial claim - Findingown data
In MMLU and CRUX, applying a larger weight to better-performing individual models tends to increase the R^2 values.
“Our analysis shows that in MMLU and CRUX, applying a larger weight to betterperforming individual models tends to increase the R 2 values.”
Quote in the paperNot a substantial claim - Findingown data
For CRUX and MATH, the gap between the effect of quality and diversity is even more pronounced.
“In particular, for CRUX and MATH, the gap between these two measures is even more pronounced.”
Quote in the paperSubstantial claim - Findingown data
Self-MoA outperforms traditional Mixed-MoA strategies in many popular benchmarks, particularly when the proposer model quality varies.
“Our experiments demonstrate that Self-MoA outperforms traditional Mixed-MoA strategies in many popular benchmarks, particularly when the proposer model quality varies.”
Quote in the paperSubstantial claim - Findingown data
The variance of models on CRUX is generally higher than that of the other two tasks.
“The results in Table 3 indicate that the variance of models on CRUX is generally higher than that of the other two tasks, which could bias the average performance towards CRUX.”
Quote in the paperSubstantial claim - Findingnone given
The trends among MMLU, CRUX, and MATH are consistently aligned.
“The trends among MMLU, CRUX, and MATH are consistently aligned.”
Quote in the paperNot a substantial claim - Findingown data
The two top models significantly outperform Qwen1.5-110B-Chat.
“Qwen1.5-110B-Chat is not used as the aggregator since the two top models significantly outperform it.”
Quote in the paperSubstantial claim - Findingown data
Using GPT-4o as the aggregator achieves the best performance on MT-Bench even with fewer forward passes compared to 3-Layer MoA with GPT-4o.
“Table 10 shows that Self-MoA outperforms its Mixed-MoA counterpart, and using GPT-4o as the aggregator can achieve the best performance even with fewer forward passes compared to 3-Layer MoA with GPT-4o.”
Quote in the paperSubstantial claim - Findingown data
USC performs worse than its MoA counterpart on AlpacaEval 2.0 when proposers and aggregators are controlled.
“We report the result in Table 12, which shows that USC performs worse than its MoA counterpart when proposers and aggregators are controlled.”
Quote in the paperSubstantial claim - Interpretationown data
The higher variance on CRUX could bias the average performance towards CRUX.
“The results in Table 3 indicate that the variance of models on CRUX is generally higher than that of the other two tasks, which could bias the average performance towards CRUX.”
Quote in the paperSubstantial claim - Interpretationreasoning
Self-MoA effectively addresses the quality-diversity trade-off by aggregating outputs from a single high-performing model.
“By aggregating outputs from a single high-performing model, Self-MoA effectively addresses the quality-diversity trade-off.”
Quote in the paperSubstantial claim - Interpretationreasoning
A worse proposer model will lead to suboptimal performance of Self-MoA.
“Therefore, a worse proposer model will lead to suboptimal performance of Self-MoA.”
Quote in the paperSubstantial claim - Interpretationreasoning
Self-MoA-Seq can handle an unlimited number of responses regardless of context length constraints.
“Self-MoA-Seq uses a sliding window to aggregate a fixed number of responses at a time, allowing it to handle an unlimited number of responses, regardless of context length constraints.”
Quote in the paperSubstantial claim - Interpretationnone given
Ensembling performance is more sensitive to quality rather than diversity.
“As we shall see later in Section 4, ensembling performance is more sensitive to quality rather than diversity.”
Quote in the paperSubstantial claim - Interpretationown data
Self-MoA over one strong model is effective.
“In previous sections, we have provided evidence that Self-MoA over one strong model is straightforward but effective.”
Quote in the paperSubstantial claim - Interpretationreasoning
Self-MoA-Seq can extend the effectiveness of Self-MoA to LLMs with shorter context lengths without sacrificing performance.
“These findings suggest that Self-MoA-Seq can extend the effectiveness of Self-MoA to LLMs with shorter context lengths, without sacrificing performance.”
Quote in the paperSubstantial claim- Rests on an assumption: Every LLM has a context length limit (e.g., 8192 tokens for Gemma 2), which restricts the number of responses an aggregator can process at once.
- Interpretationreasoning
Self-MoA cannot scale indefinitely by simply increasing the computation budget.
“Intuitively, Self-MoA cannot scale indefinitely by simply increasing the computation budget for at least three reasons:”
Quote in the paperSubstantial claim- Rests on an assumption: As more responses are sampled from a single model, the diversity among those samples tends to plateau.
- Rests on an assumption: Aggregating information from many samples is more challenging for LLMs compared to handling a smaller number of samples.
- Rests on an assumption: Every LLM has a context length limit (e.g., 8192 tokens for Gemma 2), which restricts the number of responses an aggregator can process at once.
- Interpretationreasoning
The quality-diversity trade-off helps explain why Self-MoA achieves superior performance across various benchmarks.
“This trade-off between diversity and quality helps to explain why Self-MoA achieves superior performance across various benchmarks.”
Quote in the paperSubstantial claim - Interpretationown data
Rather than finding the most consistent response, MoA and Self-MoA can encourage LLMs to synthesize the references and produce a better response.
“This further suggests that rather than finding the most consistent response, MoA and Self-MoA can encourage LLM to synthesize the references and produce a better response.”
Quote in the paperSubstantial claim - Interpretationnone given
Self-MoA utilizes in-model diversity to enhance the performance of large language models during inference.
“Self-MoA, an innovative approach that utilizes in-model diversity to enhance the performance of large language models during inference.”
Quote in the paperSubstantial claim - Interpretationown data
Cross-model diversity can benefit MoA.
“This result provides another example of cross-model diversity benefiting MoA”
Quote in the paperSubstantial claim - Interpretationown data
There is no universal compute-optimal solution for the number of samples.
“Adding more samples can have both positive and negative effects, meaning there is no universal compute-optimal solution.”
Quote in the paperSubstantial claim - Interpretationown data
Self-MoA is still a very competitive baseline under the constructed mixture task setting.
“Given the reported small margin, we argue that Self-MoA is still a very competitive baseline under this setting”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 12
- May combine two claims. 18
- The quote only partly states this claim. 1
Classified as probably not a substantial claim. May combine two claims.
PremisecitationIn tasks with automatic verifiers available, such as math and code, simply sampling LLMs multiple times can significantly improve the pass@k metric and boost the success rate of solving the tasks.
“In tasks with automatic verifiers available, such as math [Hendrycks et al., 2021] and code [Chen et al., 2021], simply sampling LLMs multiple times can significantly improve the pass@k metric and hence boost the success rate of solving the tasks”
Quote in the paperSubstantial claim2 of 2 sources checked- cites Hendrycks et al., 2021 source
Section 3.1 (p. 5) says MATH answers can be assessed automatically by comparing a model’s boxed answer with the ground truth. The cited paper therefore supports math as an example of a task with automatic answer checking, but our copy does not show that sampling a model multiple times improves pass@k or task-solving success. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites Chen et al., 2021 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationRepeated sampling is recently regarded as one approach of scaling compute during inference time.
“Therefore, repeated sampling is recently regarded as one approach of scaling compute during inference time [Brown et al., 2024].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Brown et al., 2024 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIn more general tasks without verification tools, techniques like majority vote, self-consistency, and best-of-n can choose the most promising response from candidate responses.
“In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency, and best-of-n to choose the most promising one from candidate responses”
Quote in the paperNot a substantial claim1 of 3 sources checked- cites Wang et al., 2022 source
Section 2 (pp. 2–3) describes sampling candidate reasoning paths and choosing the most consistent final answer by majority vote, without a separate verifier. It does not establish best-of-n selection, and it limits the described method to tasks with answers from a fixed set rather than general response-selection tasks. (LLM, conf=0.86)
Experimental — accuracy not yet measured - cites Chen et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Gui et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe length-controlled (LC) win rate is a more robust evaluation metric for model-based evaluation.
“To address length bias inherent in model-based evaluation, Dubois et al. [2024] introduced the length-controlled (LC) win rate as a more robust evaluation metric.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Dubois et al., 2024 source
May combine two claims. The quote only partly states this claim.
Findingown dataThe coefficient for quality (α) is greater than the coefficient for diversity (β) across all three datasets, indicating MoA's performance is more sensitive to quality than diversity.
“we observe that α is greater than β across all three datasets.”
- Also stated in: § 4.1 Statistical Analysis · hedged
“These results suggest that MoA's performance is particularly sensitive to variations in quality”
Quote in the paperSubstantial claim- Also stated in: § 4.1 Statistical Analysis · hedged
May combine two claims.
Findingown dataMoA performance has a positive correlation with both quality and diversity.
“we confirm that MoA performance has a positive correlation with both quality and diversity.”
- Also stated in: § 4 The Quality-Diversity Trade-off · plain
“When the quality is fixed, increasing diversity can enhance MoA's performance.”
- Also stated in: § 4 The Quality-Diversity Trade-off · plain
“When the diversity is fixed, improving quality can also boost MoA's performance.”
Quote in the paperSubstantial claim- Also stated in: § 4 The Quality-Diversity Trade-off · plain
May combine two claims.
Findingown dataSelf-MoA-Seq performs at least as effectively as Self-MoA, enabling scalable ensembling for LLMs with shorter context lengths without compromising final performance.
“Our findings show that Self-MoA-Seq performs at least as effectively as Self-MoA, enabling scalable ensembling for LLMs with shorter context lengths without compromising final performance.”
- Also stated in: § Abstract · plain
“sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.”
Quote in the paperSubstantial claim- Also stated in: § Abstract · plain
May combine two claims.
FindingreasoningMoA with iimmdd surpasses mmmddd on MMLU but underperforms on CRUX and MATH.
“while MoA with iimmdd surpasses mmmddd on MMLU, it underperforms on CRUX and MATH.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataQwen2-7B-Instruct, DeepSeek-Coder-V2-Lite-Instruct, and Qwen2-Math-7B-Instruct excel on MMLU, CRUX, and MATH, respectively.
“As shown in Table 3, Qwen2-7B-Instruct, DeepSeek-Coder-V2-Lite-Instruct, and Qwen2-Math-7B-Instruct excel on MMLU, CRUX, and MATH, respectively.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataResults on MT-Bench align with findings from AlpacaEval 2.0, reinforcing the effectiveness of Self-MoA.
“The results align with our findings from AlpacaEval 2.0, reinforcing the effectiveness of Self-MoA.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe p-values for both α and β are less than 0.001, indicating a significant correlation between MoA's performance and both quality and diversity.
“We observe that the p-values for both α and β are less than 0.001, indicating a significant correlation between MoA's performance and both quality and diversity”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Arnold, 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Findingown dataSelf-MoA outperforms the 3-Layer Mixed-MoA baseline with only half the forward passes.
“Notably, Self-MoA outperforms the 3-Layer Mixed-MoA baseline with only half the forward passes.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenSelf-MoA-Seq delivers performance comparable to or slightly better than Self-MoA.
“Self-MoA-Seq delivers performance that is comparable to, or slightly better than, Self-MoA.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataBoth Self-MoA and Self-MoA-Seq significantly improve performance over the individual base model.
“Both Self-MoA and Self-MoA-Seq significantly improve performance over the individual base model.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationPrevious studies have shown that combining model outputs from repeated sampling can yield a better response in various domains.
“Previous studies have shown that combining model outputs from repeated sampling can yield a better response in various domains.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationPrevious research highlights the significance of model diversity within the proposers for optimizing the performance of MoA.
“Previous research highlights the significance of model diversity within the proposers for optimizing the performance of MoA, primarily focusing on strategies for ensembling a diverse set of individual models.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Wang et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Zhang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisecitationA single model with carefully designed prompts can sometimes match the performance of agent discussions.
“Interestingly, Wang et al. [2024b] shows that a single model with carefully designed prompts can sometimes match the performance of agent discussions.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisecitationAgent discussions mainly outperform a single LLM when the prompts are insufficient.
“Moreover, agent discussions mainly outperform a single LLM when the prompts are insufficient.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMoA has garnered significant interest and achieves superior performance in challenging tasks such as instruction following, summarization, data extraction, and real-world code issue resolution.
“Among various ensembling approaches, Mixture-of-Agents (MoA) [Wang et al., 2024a] has garnered significant interest, achieving superior performance in challenging tasks such as instruction following [Wang et al., 2024a], summarization, data extraction [OpenPipe, 2024], and real-world code issue resolution [Zhang et al., 2024b].”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Wang et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites OpenPipe, 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Zhang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Interpretationown dataWhile using multiple models intuitively offers greater diversity, ensembling multiple outputs from a single model is more effective.
“This suggests that, while using multiple models intuitively offers greater diversity, ensembling multiple outputs from a single model is more effective.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown dataThe 2-Layer experiments are fair comparisons because the authors report their implemented results for the other experiments and copy the numbers from Wang et al. [2024a] for 3-Layer MoA settings.
“We copy the numbers from Wang et al. [2024a] for 3-Layer MoA settings, and report our implemented results for the other experiments to ensure that 2-Layer experiments are fair comparisons.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wang et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningSelf-MoA-Seq unlocks LLMs that are constrained by the context length and enables computation scaling during inference.
“Self-MoA-Seq unlocks LLMs that are constrained by the context length and enables computation scaling during inference.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenSelf-MoA asks LLMs to synthesize outputs generated from repeated sampling, hence can further improve over each individual output.
“Unlike majority vote or best-of-N, Self-MoA asks LLMs to synthesize outputs generated from repeated sampling, hence can further improve over each individual output.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataSimply aiming for greater diversity in the proposer mixture can result in lower overall quality, which may negatively impact MoA's performance.
“However, as shown in Figure 2, compared to Self-MoA on the best-performing model, simply aiming for greater diversity in the proposer mixture can result in lower overall quality, which may negatively impact MoA's performance.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
InterpretationreasoningLeveraging outputs from the best-performing individual model can more effectively navigate the quality-diversity trade-off by creating a higher-quality proposer mixture.
“Intuitively, leveraging outputs from the best-performing individual model can more effectively navigate the quality-diversity trade-off by creating a higher-quality proposer mixture.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataAssigning weights to the three tasks based on the inverse of their variance ensures that each task contributes equally to the overall performance metric.
“To ensure that each task contributes equally to the overall performance metric, we assign weights to the three tasks based on the inverse of their variance.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe finding that MoA is sensitive to quality naturally explains the effectiveness of Self-MoA, as it utilizes the strongest model as the proposer, ensuring high quality in its outputs.
“This finding naturally explains the effectiveness of Self-MoA, as it utilizes the strongest model as the proposer, ensuring high quality in its outputs.”
Quote in the paperSubstantial claim
The authors state this as an assumption.
The aggregator can select the correct answer from the proposers, particularly when the responses of individual models are inconsistent.
“It is reasonable to assume that the aggregator can select the correct answer from the proposers, particularly when the responses of individual models are inconsistent.”
The authors state this as an assumption.
Without loss of generality, the individual model accuracies can be ordered q1 ≥ q2 ≥ ... ≥ q6.
“without loss of generality, we assume q 1 ≥ q 2 ≥ . . .≥ q 6 .”
The paper relies on this without support.
As more responses are sampled from a single model, the diversity among those samples tends to plateau.
“As more responses are sampled from a single model, the diversity among those samples tends to plateau.”
The paper relies on this without support.
Aggregating information from many samples is more challenging for LLMs compared to handling a smaller number of samples.
“Aggregating information from many samples is more challenging for LLMs compared to handling a smaller number of samples.”
The paper relies on this without support.
Every LLM has a context length limit (e.g., 8192 tokens for Gemma 2), which restricts the number of responses an aggregator can process at once.
“Every LLM has a context length limit (e.g., 8192 tokens for Gemma 2), which restricts the number of responses an aggregator can process at once.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table 3 caption reverses the model labels
MinorThe caption assigns m to DeepSeek-Coder and d to Qwen2-Math, whereas the methods section assigns d to DeepSeek-Coder and m to Qwen2-Math. The conflicting labels make the table's proposer configurations ambiguous.
“The labels i, m, and d refer to Qwen2-7B-Instruct, DeepSeek-Coder-V2-Lite-Instruct, and Qwen2-Math-7B-Instruct, respectively.”
“For convenience, we denote Qwen2-7B-Instruct as i, DeepSeek-Coder-V2-Lite-Instruct as d, and Qwen2-Math-7B-Instruct as m.”
Claimed MMLU R2 at K=2 conflicts with Table 5
MinorThe prose says Centered-1/K-Norm at K=2 exceeds 0.9 on MMLU. Table 5 gives 0.832 for that setting; the value exceeding 0.9 appears at K=4.
“we find that the R 2 value can exceed 0.9 with K = 2 using the Centered-1/K-Norm.”
“| MMLU | K-Norm Centered-1/K-Norm | 0.771 0.771 | 0.809 0.832 0.845 0.881 0.902 0.903 | |”
Cites
Introduction to the Practice of Statistics
Thunder, M. (1990) · Mathematical Gazette · doi:10.2307/3620137 · Cited by 3,281 · cited 1×
“We observe that the p-values for both α and β are less than 0.001, indicating a significant correlation between MoA’s performance and both quality and diversity (Arnold, 1990).”
InfluentialLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1× · see citing works
“5-72B-Chat (Bai et al., 2023), WizardLM-8x22B (Xu et al., 2023), LLaMA-3-70B-Instruct (Touvron et al., 2023), Mixtral-8x22B-Instruct-v0.”
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1× · see citing works
“In tasks with automatic verifiers available, such as math (Hendrycks et al., 2021) and code (Chen et al., 2021), simply sampling LLMs multiple times can significantly improve the pass@k metric and hence boost the success rate of solving the tasks (Roziere et al., 2023; Li et al., 2022; Brown et…”
WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsPreprint
Xu, C., Sun, Q., Zheng, K., Geng, X.-B., Zhao, P., Feng, J., Tao, C., Lin, Q.-W., & Jiang, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2304.12244 · Cited by 1,362 · cited 3× · see citing works
“5-72B-Chat (Bai et al., 2023), WizardLM-8x22B (Xu et al., 2023), LLaMA-3-70B-Instruct (Touvron et al., 2023), Mixtral-8x22B-Instruct-v0.”
InfluentialCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1× · see citing works
“…with automatic verifiers available, such as math (Hendrycks et al., 2021) and code (Chen et al., 2021), simply sampling LLMs multiple times can significantly improve the pass@k metric and hence boost the success rate of solving the tasks (Roziere et al., 2023; Li et al., 2022; Brown et al., 2024).”
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 3× · see citing works
“We conduct further experiments to compare Self-Consistency (Wang et al., 2022) with MoA and Self-MoA on the AlpacaEval 2.0 benchmark.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2019 · Cited by 755 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1× · see citing works
“…with automatic verifiers available, such as math (Hendrycks et al., 2021) and code (Chen et al., 2021), simply sampling LLMs multiple times can significantly improve the pass@k metric and hence boost the success rate of solving the tasks (Roziere et al., 2023; Li et al., 2022; Brown et al., 2024).”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning
Friedman, D., & Dieng, A. B. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2210.02410 · Cited by 397 · cited 4×
“We measure the quality and diversity as follows: • Diversity : We utilize the Vendi Score (Dan Friedman & Dieng, 2023) to assess the diversity among individual models in the proposer mixture.”
InfluentialBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · cited 15× · see citing works
“For each benchmark, we use the best-performing model as both the proposer and aggregator (Qwen2-7B-Instruct for MMLU and DeepSeek-Coder-V2-Lite-Instruct for CRUX), with a sampling temperature of 0.7.”
InfluentialEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.19118 · Cited by 320 · see citing works
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 13× · see citing works
“In particular, for CRUX and MATH, the gap between these two measures is even more pronounced.”
InfluentialMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 2× · see citing works
“We conduct evaluations across a diverse set of benchmarks: • MMLU (Hendrycks et al., 2020) is a multiple-choice dataset designed to assess a model’s multitask accuracy.”
Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?Preprint
Wang, Q., Wang, Z., Su, Y., Tong, H.-H., & Song, Y.-Q. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.18272 · Cited by 243 · cited 1× · see citing works
“For example, Stechly et al. (2023), Valmeekam et al. (2023) Wang et al. (2024b) shows that a single model with carefully designed prompts can sometimes match the performance of agent discussions.”
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language ModelsPreprint
Lu, K.-M., Yuan, H., Lin, R.-J., et al. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.08692 · Cited by 172 · cited 1× · see citing works
“…explored various methods to leverage a diverse set of models, including but not limited to prompting (Wang et al., 2024a), weight averaging (Lin et al., 2024; Ramé et al., 2024), routing (Jiang et al., 2024b; Lu et al., 2023), training a generative fusion model (Jiang et al., 2023b), and so on.”
Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.00118 · Cited by 149 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · cited 1× · see citing works
“1 (Jiang et al., 2024a), and dbrx-instruct (Team et al., 2024b).”
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Jiang, D., Ren, X., & Lin, B. Y. (2023) · arXiv · doi:10.48550/arxiv.2306.02561 · Cited by 141 · cited 1× · see citing works
“Previous work on ensembles (Wang et al., 2024a; Jiang et al., 2023a; Zhang et al., 2024b) primarily focuses on increasing the diversity of models within the proposer mixture.”
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n SamplingPreprint
Garbacea, C., Gui, L., & Veitch, V. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.00832 · Cited by 139 · cited 3× · see citing works
“…have explored various methods to leverage a diverse set of models, including but not limited to prompting (Wang et al., 2024a), weight averaging (Lin et al., 2024; Ramé et al., 2024), routing (Jiang et al., 2024b; Lu et al., 2023), training a generative fusion model (Jiang et al., 2023b), and…”
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · see citing works
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Chen, J. C.-Y., Saha, S., & Bansal, M. (2023) · arXiv · doi:10.48550/arxiv.2309.13007 · Cited by 64 · see citing works
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10671 · Cited by 63 · see citing works
WPO: Enhancing RLHF with Weighted Preference OptimizationPreprint
Zhou, W., Agrawal, R., Zhang, S., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.11827 · Cited by 56 · cited 1× · see citing works
“To further validate the effectiveness of Self-MoA, we apply it to the two top-performing models on AlpacaEval 2.0: gemma-2-9b-it-WPO-HB (Zhou et al., 2024) and gemma-2-9b-it-SimPO (Meng et al., 2024).”
InfluentialDeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
DeepSeek-AI, Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.11931 · Cited by 51 · cited 8× · see citing works
“For each benchmark, we use the best-performing model as both the proposer and aggregator (Qwen2-7B-Instruct for MMLU and DeepSeek-Coder-V2-Lite-Instruct for CRUX), with a sampling temperature of 0.7.”
InfluentialWARP: On the Benefits of Weight Averaged Rewarded Policies
Ramé, A., Ferret, J., Vieillard, N., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.16768 · Cited by 48 · cited 1×
“…explored various methods to leverage a diverse set of models, including but not limited to prompting (Wang et al., 2024a), weight averaging (Lin et al., 2024; Ramé et al., 2024), routing (Jiang et al., 2024b; Lu et al., 2023), training a generative fusion model (Jiang et al., 2023b), and so on.”
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27 · cited 1× · see citing works
“As the community is becoming more aware of scaling inference time computing (Brown et al., 2024; Snell et al., 2024; Wu et al., 2024), one natural question to ask is: Given a strong model, does Self-MoA’s performance scale with the number of repeated samples?”
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 25 · cited 9× · see citing works
“Previous studies have explored various methods to leverage a diverse set of models, including but not limited to prompting (Wang et al., 2024a), weight averaging (Lin et al., 2024; Ramé et al., 2024), routing (Jiang et al., 2024b; Lu et al., 2023), training a generative fusion model (Jiang et al.,…”
InfluentialLi, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05120 · Cited by 22 · cited 1× · see citing works
“In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency, and best-of-n to choose the most promising one from candidate responses (Wang et al., 2022; Chen et al., 2023b; Gui et al., 2024; Li et al., 2024).”
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024) · arXiv · doi:10.48550/arxiv.2404.04475 · Cited by 17 · cited 2× · see citing works
“We adopt the same experiment setting as Wang et al. (2024a) in AlpacaEval 2.0 benchmark (Dubois et al., 2024) and compare the performance of Mixed-MoA and Self-MoA 1 .”
InfluentialLarge Language Monkeys: Scaling Inference Compute with Repeated Sampling
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024) · arXiv · doi:10.48550/arxiv.2407.21787 · Cited by 14 · cited 3× · see citing works
“As the community is becoming more aware of scaling inference time computing (Brown et al., 2024; Snell et al., 2024; Wu et al., 2024), one natural question to ask is: Given a strong model, does Self-MoA’s performance scale with the number of repeated samples?”
Universal Self-Consistency for Large Language Model Generation
Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2311.17311 · Cited by 13 · cited 2× · see citing works
“We report the result in Table 13, which shows that USC performs worse than its MoA counterpart when proposers and aggregators are controlled.”
SimPO: Simple Preference Optimization with a Reference-Free Reward
Meng, Y., Xia, M., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2405.14734 · Cited by 12 · cited 1× · see citing works
“To further validate the effectiveness of Self-MoA, we apply it to the two top-performing models on AlpacaEval 2.0: gemma-2-9b-it-WPO-HB (Zhou et al., 2024) and gemma-2-9b-it-SimPO (Meng et al., 2024).”
InfluentialSelf-citationSarjana, K., Hayati, L., & Wahidaturrahmi, W. (2021) · doi:10.20414/betajtm.v13i2.390 · Cited by 12
GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems
Stechly, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.12397 · Cited by 11 · cited 1× · see citing works
“For example, Stechly et al. (2023), Valmeekam et al. (2023) Wang et al. (2024b) shows that a single model with carefully designed prompts can sometimes match the performance of agent discussions.”
Mitigating the Alignment Tax of RLHF
Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.06256 · Cited by 10 · cited 2× · see citing works
“…have explored various methods to leverage a diverse set of models, including but not limited to prompting (Wang et al., 2024a), weight averaging (Lin et al., 2024; Ramé et al., 2024), routing (Jiang et al., 2024b; Lu et al., 2023), training a generative fusion model (Jiang et al., 2023b), and…”
Self-citationCan Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 8 · cited 1× · see citing works
“For example, Stechly et al. (2023), Valmeekam et al. (2023) Wang et al. (2024b) shows that a single model with carefully designed prompts can sometimes match the performance of agent discussions.”
Towards Building Specialized Generalist AI with System 1 and System 2 FusionPreprint
Zhang, K., Qi, B., & Zhou, B.-W. (2024) · arXiv.org · doi:10.48550/arxiv.2407.08642 · Cited by 8 · see citing works
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., & Wang, S. I. (2024) · arXiv · doi:10.48550/arxiv.2401.03065 · Cited by 7 · cited 15× · see citing works
“For each benchmark, we use the best-performing model as both the proposer and aggregator (Qwen2-7B-Instruct for MMLU and DeepSeek-Coder-V2-Lite-Instruct for CRUX), with a sampling temperature of 0.7.”
InfluentialWu, Y., Sun, Z., Li, S., Welleck, S., & Yang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.00724 · Cited by 7 · cited 1× · see citing works
“As the community is becoming more aware of scaling inference time computing (Brown et al., 2024; Snell et al., 2024; Wu et al., 2024), one natural question to ask is: Given a strong model, does Self-MoA’s performance scale with the number of repeated samples?”
MoA is All You Need: Building LLM Research Team using Mixture of AgentsPreprint
Chen, S., Zeng, L., Raghunathan, A., et al. (2024) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2409.07487 · Cited by 6 · see citing works
Gema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04127 · Cited by 1 · cited 2× · see citing works
“We adopt MMLU-redux (Gema et al., 2024) for evaluation, which is a subset of MMLU with 3,000 samples fixing the errors in the dataset through human re-annotating.”
Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
Zhang, K., Yao, W., Liu, Z., Feng, Y., Liu, Z., Murthy, R., Lan, T., Li, L., Lou, R., Xu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.07060 · Cited by 1 · cited 3× · see citing works
“However, while existing MoA focuses on unleashing the strength from multiple different models (Wang et al., 2024a; Jiang et al., 2023b; Zhang et al., 2024b), we demonstrate the trade-off between diversity and quality within the proposers, highlighting that focusing solely on diversity may…”
InfluentialOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 4×
“Large language models have made remarkable strides in improving performance across different domains, with notable examples such as GPT (Achiam et al., 2023), Gemini (Team et al., 2023), and Claude (An-thropic, 2023).”
Influential
Flagged first, then by citations
Research works printed but not linked: 2
- A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024b. URL https://arxiv.org/abs/2401.04088. · arXiv
- D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion, 2023b. URL https://arxiv.org/abs/2306.02561. · arXiv
Cited by
Sorted by citations
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
Zhang, Y.-Q., Li, H., Wang, C.-X., Chen, L.-Y., Zhang, Q.-S., Ye, P., Feng, S., Wang, D.-L., Wang, Z., Wang, X.-R., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.19797 · Cited by 41
Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM ReasoningPreprint
Eo, S., Moon, H., Zi, E. H., Park, C., & Lim, H.-J. (2025) · arXiv.org · doi:10.48550/arxiv.2504.05047 · Cited by 33 · see citing works
“Additionally, iterative debates may propagate errors or introduce a trade-off between diversity and quality (Wang et al., 2024; Kapoor et al., 2024; Li et al., 2025).”
Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills
Chen, J., Yun, S., Stengel-Eskin, E., et al. (2025) · doi:10.48550/arxiv.2503.05641 · Cited by 28
“Similarly, Self-MoA (Li et al., 2025) suggest that optimal performance can be achieved by invoking the task-best model multiple times alongside the task-best aggregator.”
InfluentialGraph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration
Yun, S., Peng, J., Li, P.-Z., Fan, W.-D., Chen, J., Zou, J. Y., Li, G.-H., & Chen, T.-L. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2604.17148 · Cited by 26
“Multi-agent baselines : Debate (Du et al., 2023b), Self-Consistency (SC) (Wang et al., 2023a), Refine (Madaan et al., 2023), ReConcile (Chen et al., 2023), MoA (Wang et al., 2024a), and Self-MoA (Li et al., 2025).”
InfluentialVeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generation
Ping, H., Bhattacharjee, A., Zhang, P.-Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.27617 · Cited by 16
“Recent analysis (Li et al., 2025b) reveals that MoA performance correlates strongly with both quality and diversity of the proposers, with quality showing higher correlation.”
EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering
Huang, J.-T., Zhang, Z.-Y., Shi, K.-W., Ye, Y.-F., & Zhang, C.-X. (2026) · arXiv.org · doi:10.48550/arxiv.2604.05149 · Cited by 17
“…✗ Search MAPRO (Zhang et al., 2026) Multi-agent ✗ ✓ ✗ Inference HiveMind (Xia et al., 2025) Multi-agent System Design OPTIMA (Chen et al., 2025a) Multi-agent (Wang et al., 2025a) Multi-agent ✗ ✗ ✗ Inference Self-MoA (Li et al., 2026a) Single-agent ✗ ✗ ✗ Inference ADAS (Hu et al., 2025) Multi-agent”
TUMIX: Multi-Agent Test-Time Scaling with Tool-Use MixturePreprint
Chen, Y., CHEN, J.-C., Meng, R., Yin, J., Li, N., Fan, C., Wang, C., Pfister, T., & Yoon, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.01279
iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference
Fan, W., Yoon, J., & Ji, B. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2511.11306 · Cited by 12
“Some methods generate multiple outputs for joint evaluation (e.g., CoMM (Chen, Han, and Zhang 2024)), while others assign hierarchical agent roles (e.g., Mixture-of-Agents (MoA) to progressively re-fine reasoning (Wang et al. 2025; Chen et al. 2024; Li et al. 2024b)).”
Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
Fein-Ashley, J., Parikh, D., Kannan, R., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.21164 · Cited by 12
“…routing to one or a few experts [6, 35, 63, 39, 15, 20, 28, 26, 60, 58, 23], sometimes with ensemble sampling [58]; (ii) response-level collaboration (mixture-of-agents) [48, 31, 33, 4]; and (iii) parameter-space fusion (merging, continual learning, distillation) [51, 55, 14, 21, 59, 52, 47].”
Chen, J. (2026) · arXiv.org · doi:10.48550/arxiv.2606.27288 · Cited by 11
“…sb , so our small-G claim is conservative, though it flatters the learned-router comparison), random and random-mixing-at-matched-budget, cost-matched Self-MoA [24], a partition-conditioned oracle and a partition-free per-query oracle, a confidence cascade, majority vote, and heterogeneous fusion.”
Kim, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.20768 · Cited by 9
SANet: A Semantic-Aware Agentic AI Networking Framework for Cross-Layer Optimization in 6G
Xiao, Y., Li, X., Zhou, H., et al. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3691804 · Cited by 7
“In this paper, we propose an AgentNet-based solution in which the user’s semantic goal and demand can be predicted and proactively addressed by a carefully selected subset of collaborative agents.”
Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference
Guo, X., Wang, S., Ji, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.07571 · Cited by 7
“Building on this, variants such as sparse MoA (Fu et al., 2024) and Self MoA (Li et al., 2025) have been introduced.”
Lian, D.-C., Huang, R.-S., Chen, P.-E., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.16809 · Cited by 7
“Because of the difference between the settings (e.g., Large Reasoning Models vs. LLMs), there is no strong evidence to suggest that the trend will follow past MoA discussions such as Li et al. (2025).”
Self-citationMEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation
Yu, S., Zhang, Y., Wang, Z., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2506.17113 · Cited by 7
“…recent advances have explored leveraging a mixture of expert models (or agents ) (Li et al., 2024b; Wang et al., 2024; Chen et al., 2025; Li et al., 2025; Cao et al., 2025) that combine independently pre-trained models across diverse knowledge domains, moving beyond the traditional…”
Yu, F.-C., Yao, J.-C., Wang, Z., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.24855 · Cited by 6
“When self-consistency among candidates is low, Self-MoA may merge spurious reasoning paths, introducing additional errors.”
InfluentialContextual Drag: How Errors in the Context Affect LLM Reasoning
Cheng, Y., Zhu, X., Zhao, H.-Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.04288 · Cited by 5
“Our criteria for constructing the evaluation set for contextual drag is to avoid confounding performance drops caused by trivial, low-quality in-context drafts observed by Li et al. (2025c).”
InfluentialDFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance
Cohen, S., Goldshlager, N., Cohen-Inger, N., Shapira, B., & Rokach, L. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.17479 · Cited by 5
“At the other extreme, Self-MoA shows that repeatedly sampling a single strong model can outperform mixtures when one model dominates the quality spectrum (Li et al., 2025).”
A Unifying Perspective on Model Reuse: From Small to Large Pre-Trained Models
Zhou, D.-W., & Ye, H. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/1201 · Cited by 5
“Lastly, modules of different PTMs can also collaborate together to enhance prediction robustness [Wang et al. , 2024; Li et al. , 2025].”
MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs
Mitra, P., & Ulukus, S. (2025) · International Conference on Machine Learning and Applications · doi:10.1109/icmla66185.2025.00140
“Works by [7], [8] showed that using a single LLM with high temperature creates diverse answers to put in a long context for better collaborative inference.”
When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling
Yun, H., Ki, K., Lee, J.-H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.15346 · Cited by 3
“Within this framework, Self-MoA (Li et al., 2025) shows that using a single best-performing model can outperform using multiple distinct models in some cases, while Symbolic-MoE (Chen et al., 2025a) introduces query-adaptive model routing.”
Multi-Agent Collaborative Intelligence: Dual-Dial Control for Reliable LLM Reasoning
Chang, E. Y., & Chang, E. Y. (2025) · arXiv.org · doi:10.48550/arxiv.2510.04488 · Cited by 3
“Mixture-2 of-Agents [Wang et al., 2024a, Li et al., 2025] improves performance via layered ensembles but under-specifies convergence beyond heuristic rounds.”
Making, not Taking, the Best of NPreprint
Khairi, A., D’souza, D., Fadaee, M., & Kreutzer, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00931
“For the more diverse generative evaluation scenarios that we are targeting, however, we find that FusioN is fairly robust with respect to the teacher pool (section 5), and sampling from a single teacher—the proposed solution by Li et al. [2025a]—performs significantly worse.”
ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scaling
Ping, H., Bhattacharjee, A., Zhang, P.-Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.24437 · Cited by 2
“Within information consolidation , beyond RMoA (Xie et al., 2025), SMoA (Li et al., 2025a) sparsifies agent interactions via dedicated judge and moderator agents, while Self-MoA (Li et al., 2025b) instead argues quality dominates diversity, aggregating samples from a single strong model.”
Mixture of Complementary Agents for Robust LLM Ensemble
Zhang, Y.-C., Lu, K., Zhang, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.24048 · Cited by 2
“This observation is supported by prior work (Li et al., 2025) and our experiments in Section B.2.”
InfluentialAn Inspectable LLM Council for Multi-Model Research Answer Aggregation
Wu, S., Li, X., Feng, Y., Li, Y.-F., Wang, Z., & Wang, R. (2026) · doi:10.48550/arxiv.2604.02923 · Cited by 2
“Language Model Council instead studies collective generation and evaluation of benchmark tasks [13].”
Zhang, T., Liang, S., Zheng, W.-M., et al. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3667846 · Cited by 2
“Previous works [23], [66], [67] have demonstrated that leveraging the collective expertise of multiple LLMs can lead to the development of more capable and robust models for a variety of tasks.”
Wen, J.-R., Wei, Y., Yu, X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.16596 · Cited by 2
“Self-MoA Li et al. (2025b) challenged the necessity of mixing heterogeneous models.”
SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems
Pan, L., Zhu, Y., Qiu, L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.03544 · Cited by 1
“Other multi-agent systems study collaboration, debate, and role-structured execution (Li et al., 2023; Wu et al., 2023; Du et al., 2024; Wang et al., 2024; Li et al., 2025).”
Conflict-Aware Multi-Agent Reasoning via Signed Graph Modeling
He, L.-G., He, L.-Z., Peng, S.-H., He, D.-J., & Li, C.-Z. (2026) · arXiv.org · doi:10.48550/arxiv.2605.19418 · Cited by 1
“For multi-agent approaches, we select MoA [22], Self-MoA [23], Complete Graph, Random Graph, DyLAN [54], AutoGen [24], GPTSwarm [25], G-Designer [27], and GoA [26].”
Impacts of Aggregation on Model Diversity and Consumer Utility
Donahue, K., & Raghavan, M. (2026) · arXiv.org · doi:10.48550/arxiv.2602.23293 · Cited by 1
“Other work studying the performance of agentic teams with specific focus on diversity includes (Kearns et al., 2026; Bedi et al., 2025; Li et al., 2025; Wang et al., 2024; Song et al., 2025).”
HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems
Yao, T., Li, Z., & Shen, Z. (2026) · arXiv.org · doi:10.48550/arxiv.2602.20229 · Cited by 1
Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale
Tang, S.-J., Lin, W.-H., Ye, P., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.01330 · Cited by 1
“…but is constrained by single model, aggregation can exceed individual model limits yet is vulnerable to low-quality outputs and noise (Li et al., 2025a; Wolf et al., 2025; Li et al., 2025c), while straightforward combination approaches struggle to effectively leverage their respective strengths.”
Haji, F., Bethany, M., Chiang, C.-Y. J., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2508.19359 · Cited by 1
“Self Mixture of Agents (Self-MoA) (Li et al., 2025) extends this concept by using multiple instances of the same model with different sampling parameters to generate diverse outputs.”
Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Making
Bao, L.-X., Peng, Z., Zhou, X.-F., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.13754 · Cited by 1
“Recent research has demonstrated that strategically combining complementary strengths across multiple LLMs through collaboration frameworks could significantly enhance performance on complex, knowledge-intensive tasks [18], [37], [38].”
Disentangling Models from Personas in Heterogeneous LLM Simulations
Roytburg, D., & Ippolito, D. (2026) · doi:10.48550/arxiv.2610.07535
“Work on heterogeneous multi-agent systems has studied downstream performance on verifiable tasks like question-answering (Chen et al., 2023; Feng et al., 2025) and code (Ye et al., 2025; Li et al., 2025), without considering open-domain cross-model dynamics or simulations.”
Revision-Aware Independent Agent Graphs for Dynamic Reasoning
Luo, Y.-H., Lali, S.-A., Moebel, J., et al. (2026) · doi:10.48550/arxiv.2610.01249
“We compare RIAG with eight multi-agent methods: Debate (Du et al., 2023), Self-Consistency (Wang et al., 2023), Refine (Madaan et al., 2023), ReConcile (Chen et al., 2024), Mixture-of-Agents (Wang et al., 2024b), Self-MoA (Li et al., 2025), Graph-of-Agents (Yun et al., 2026), and CIA (Wu et al., 2026).”
Towards Better Exploration in Sequential Test-Time Scaling
Rance, J., Pizzati, F., Sock, J., et al. (2026) · doi:10.48550/arxiv.2609.39632
“Meanwhile, Wang et al. (2025a) find that mixing models often underperforms the strongest constituent model, with similar negative results reported by Li et al. (2024).”
Separating Memory and Workflow Effects in Predicting Individual Answers
Qin, T.-Z., Yang, L., Debnath, R., et al. (2026) · doi:10.48550/arxiv.2609.33321
“For free-form text a model can select among (Chen et al., 2023) or synthesize (Wang et al., 2025b; Li et al., 2025) its own samples, as our selection control and fusion do; LLM-Blender adds trained ranking and fusion (Jiang et al., 2023), and trained verifiers improve selection in mathematics…”
Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?
Cakiroglu, M. O., Dalkilic, M., & Kurban, H. (2026) · doi:10.48550/arxiv.2609.32996
“Li et al. [24] motivate comparing pooled responses with repeated sampling from the best model.”
COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference
Alballa, N., Zhang, W.-X., Kharrat, S., et al. (2026) · doi:10.48550/arxiv.2609.26913
“Recent analyses show that such indiscriminate mixing can back-fire (Choi et al., 2025; Li et al., 2026).”
Zhang, Y., Xie, W., Li, X., Liao, W., Shu, T., Zhou, J., & Wang, J. (2026) · PLoS ONE · doi:10.1371/journal.pone.0357018
OrchSLM: Probing the Dynamics of Small Language Model Orchestration
Zhang, C.-X., & Yao, Y.-R. (2026) · doi:10.48550/arxiv.2609.13470
“MoA variants refine this pattern through proposer quality and diversity, sparse communication, and residual, attention, or memory-based flow [Li et al., 2025b,a, Xie et al., 2025, Wen et al., 2026, Ping et al., 2026].”
What Does Multi-Agent Debate Actually Change?Preprint
Qian, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.08016
“We build on LMC [Zhao et al., 2024], ChatEval [Chan et al., 2023], Peacemaker [Yao et al., 2025], and Self-MoA [Li et al., 2025 The takeaway, and what this paper does not claim.”
Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA
Lab, M. (2026) · arXiv.org · doi:10.48550/arxiv.2608.09819 · Cited by 2
“…which independently trained specialists contribute complementary capabilities, enabling the composed system either to outperform its strongest constituent or to solve tasks that no constituent can solve alone under a matched evaluation budget (Li et al., 2025b; Zhu et al., 2025; Li et al., 2026a).”
The Fleet Is the Model: Engineering Collective Intelligence with Fusion-MoA Pioneer R1
Yang, Z.-Y., & Hou, Y.-H. (2026) · doi:10.48550/arxiv.2609.26080
“The gains depend on model composition, communication protocol, and orchestrator quality [8], [9].”
Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning
Beliaev, V. (2026) · arXiv.org · doi:10.48550/arxiv.2607.07690
“Naively averaging or letting models agree, as in self-consistency and mixture-of-agents [Wang et al., 2023, 2024], tends to regress toward consensus and can dilute quality when the models differ in strength [Li et al., 2025], while two near-identical models gain little from one another.”
Quantum-Inspired Trace-Augmented Evidence Selection for Reasoning over Structured Hypothesis Spaces
Wynter, L., Sahoo, N., & Griffin, P. (2026) · arXiv.org · doi:10.48550/arxiv.2606.06941
“A follow-up by Li et al. [9] shows that Self-MoA —aggregating multiple outputs from the single strongest model—outperforms diverse multi-model mixtures on AlpacaEval 2.0, suggesting that answer diversity from one capable model is more valuable than architectural diversity across weaker models.”
InfluentialAdaMoA: Enhancing Mixture-of-Agents via Task-Adaptive Architecture Optimization
Lv, W., Xu, H., Xu, Y., Liu, J., & Xu, Z. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661164
“Ad-ditionally, not all agents are equally beneficial since low-quality outputs will distract subsequent agents, leading to performance degradation while consuming additional tokens [21].”
Mehrabi, N., Boroujeni, S. P. H., & Razi, A. (2026) · arXiv.org · doi:10.48550/arxiv.2606.00405
“A recent study revisiting MoA [12] found that aggregating outputs from a single strong model can sometimes outperform multi-model mixing, raising questions about whether model diversity at the text level is always beneficial.”
InfluentialSonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification
Ponnusamy, P., Sahni, S., Wang, J., & Dao, T. (2026) · arXiv.org · doi:10.48550/arxiv.2607.20475
“It also enables quality improvement strategies such as best-of-𝑁 decoding, where multiple diverse candidates are drawn and selected to enhance the final quality of the output (Wang et al., 2022; Chen et al., 2023b; Li et al., 2025a; Wang et al., 2025b).”
Agarwal, A. (2026) · arXiv.org · doi:10.48550/arxiv.2604.19049
“Mixture-of-Agents [36] aggregates heterogeneous outputs; Self-MoA [35] counters that same-model repeated sampling beats heterogeneous MoA by 6.6%.”
InfluentialPyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Khaled, A. (2026) · arXiv.org · doi:10.48550/arxiv.2602.19509
“Self-MoA [4] demonstrates that iteratively querying a single strong model can sometimes outperform diverse ensembles.”
WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate
Cherian, A., Doyle, R., Ben-Dov, E., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.02405
“Mixture-of-Agents (MoA) [42] treats ensembles as layers controlled by prompts and gating models, while Self-MoA [24] selects only top-performing outputs for downstream processing.”
Do LLMs Benefit from Self-Ensembles? A Study of Self-Mixture-of-Agents
Lee, S.-H., & Lee, D. (2025) · 2025 3rd International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11390896
CoLM: Collaborative Large Models via A Client-Server Paradigm
Huang, S.-Q., Huang, S.-D., & Zhang, H.-Y. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2511.06991
LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition
Shi, Y.-T., Yang, J.-D., Wang, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.19252
“Other approaches include self-integration through single-model multi-sample reasoning [41], or dynamic model selection via routing mechanisms, such as in FrugalGPT, to balance performance and cost [13].”
Huang, R., & Guo, W.-L. (2025) · arXiv.org · doi:10.48550/arxiv.2508.13654
“We discover a training-testing co-design phenomenon: applying any persona context (not necessarily the same type) during both training and testing consistently boosts performance.”
MAC: Masked Agent Collaboration Boosts Large Language Model Medical Decision-Making
Peng, Z., Bao, L.-X., & Yuan, Y. (2025) · doi:10.48550/arxiv.2507.21159
“…agents from a vast and heterogeneous pool of candidate LLMs. Existing approaches often select agents solely based on standalone performance metrics, such as the accuracy [7], [19] or diversity [28], [29], while neglecting critical operational trade-offs essential for real-world clinical deployment.”
InfluentialMMA: Cross-Domain Knowledge Integration via Mixture of Multi-Domain Agents
Jia, K., Li, J.-T., Liang, X.-B., Xiao, Y., Yang, Y., & Zhang, M. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.findings-emnlp.707
“(iv) Self-MoA(Li et al., 2025), in which only the best-performing model generates preliminary answers that are then aggregated using Aggregators.”
Influential
- Authors
- Li, W., Lin, Y., Xia, M., & Jin, C.
- Affiliations
- Princeton University (US)
- ORCID
- Jin, Chi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2502.00674
- OpenAlex
- W4407123922
- Cited by
- 0
- References
- 44
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- mixture-of-agents, large language models, output diversity, output quality, AlpacaEval, MMLU, MATH benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
