SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents

“To overcome the limitations of MoA and enhance the utility of multi-agent LLMs, we propose Sparse Mixture-of-Agents (SMoA), inspired by the design of sparse mixture-of-experts (SMoE) (Han et al., 2015; Shazeer et al., 2016).”

“…we use the Just-Eval (Lin et al., 2023) benchmark, which contains diverse samples from 5 alignment datasets: (1) AlpacaEval2 (Li et al., 2023c), (2) MT-Bench (Zheng et al., 2023), (3) LIMA (Zhou et al., 2024), (4) HH-RLHF-redteam (Ganguli et al., 2022) and (5) MaliciousInstruct (Huang et al.).”

“Furthermore, balancing load across experts, which is first proposed by Fedus et al. (2022), addresses issues like expert imbalance, which can lead to the underutilization of certain experts and over-specialization of others.”

“Moreover, drawing inspiration from the expert diversity principle (Lepikhin et al.; Jiang et al., 2024a) which is commonly applied in SMoE to balance workload, we assign distinct role descriptions to each LLM agent, facilitating diverse and divergent thinking in their problem-solving process.”

“The rapid development of Large Language Models (LLMs) (Brown et al., 2020; Anil et al., 2023; Gan et al., 2023; Dubey et al., 2024) in recent years has significantly advanced in a series of NLP tasks, such as question answering (Wang et al., 2022; Tong et al., 2023; Jin et al., 2024a), knowledge…”

“While several previous studies (Liu et al., 2023) propose rule-based methods to address it (e.g., stop when the output answers from most agents are the same), there is still a lack of a general and dynamic approach that can handle various tasks and applications.”

“5-72B-Chat (Bai et al., 2023), Qwen2-72B-instruct (Yang et al., 2024), WizardLM-8x22B (Xu et al., 2023), dbrx-instruct (Team et al., 2024).”

“One line of work is layer-based multi-agent methods (Liang et al., 2023; Du et al., 2023; Chan et al.)”

“…we use the Just-Eval (Lin et al., 2023) benchmark, which contains diverse samples from 5 alignment datasets: (1) AlpacaEval2 (Li et al., 2023c), (2) MT-Bench (Zheng et al., 2023), (3) LIMA (Zhou et al., 2024), (4) HH-RLHF-redteam (Ganguli et al., 2022) and (5) MaliciousInstruct (Huang et al.).”

“…2023; Dubey et al., 2024) in recent years has significantly advanced in a series of NLP tasks, such as question answering (Wang et al., 2022; Tong et al., 2023; Jin et al., 2024a), knowledge discovery (Li et al., 2024a,b; Tan et al., 2024a) and dialogue systems (Li et al., 2022; Sun et al., 2024).”

“Recently, mixture-of-agents (MoA) (Zhang et al., 2023; Wang et al., 2024b) has been devised, which uses multiple processors per layer to handle queries simultaneously for time efficiency purposes.”

“In all, SMoA performs well in identifying reasoning error (Tong et al., 2024; An et al., 2023) and filtering out unsafe content (Tan et al., 2024b; Beigi et al., 2024b) or misinformation (Jiang et al., 2024b; Yang et al., 2023; Beigi et al., 2024a) and moderating the information exchange process.”

“One line of work is layer-based multi-agent methods (Liang et al., 2023; Du et al., 2023; Chan et al.)”

“Routing techniques such as dynamic routing (Rosenbaum et al., 2018) and Hash-based routing (Roller et al., 2021) help optimize which experts are activated, ensuring efficient resource use.”

“In all, SMoA performs well in identifying reasoning error (Tong et al., 2024; An et al., 2023) and filtering out unsafe content (Tan et al., 2024b; Beigi et al., 2024b) or misinformation (Jiang et al., 2024b; Yang et al., 2023; Beigi et al., 2024a) and moderating the information exchange process.”

“This finding is also consistent with previous studies that focus on reasoning (Wang et al., 2022) and fairness (Han et al., 2021; Saun-MMAU SMoA shows greater potential for scaling up.”

“In all, SMoA performs well in identifying reasoning error (Tong et al., 2024; An et al., 2023) and filtering out unsafe content (Tan et al., 2024b; Beigi et al., 2024b) or misinformation (Jiang et al., 2024b; Yang et al., 2023; Beigi et al., 2024a) and moderating the information exchange process.”

“Routing techniques such as dynamic routing (Rosenbaum et al., 2018) and Hash-based routing (Roller et al., 2021) help optimize which experts are activated, ensuring efficient resource use.”

“5-72B-Chat (Bai et al., 2023), Qwen2-72B-instruct (Yang et al., 2024), WizardLM-8x22B (Xu et al., 2023), dbrx-instruct (Team et al., 2024).”

“The key idea in SMoE (Shazeer et al., 2016; Chen et al., 2023; He et al., 2023, 2024) is to activate only a few experts per input, significantly reducing computational costs compared to the traditional MoE models.”

“The rapid development of Large Language Models (LLMs) (Brown et al., 2020; Anil et al., 2023; Gan et al., 2023; Dubey et al., 2024) in recent years has significantly advanced in a series of NLP tasks, such as question answering (Wang et al., 2022; Tong et al., 2023; Jin et al., 2024a), knowledge…”

“In our experiment, we follow the implementation of (Wang et al., 2024b).”

“…al., 2023; Dubey et al., 2024) in recent years has significantly advanced in a series of NLP tasks, such as question answering (Wang et al., 2022; Tong et al., 2023; Jin et al., 2024a), knowledge discovery (Li et al., 2024a,b; Tan et al., 2024a) and dialogue systems (Li et al., 2022; Sun et al.,…”

“In all, SMoA performs well in identifying reasoning error (Tong et al., 2024; An et al., 2023) and filtering out unsafe content (Tan et al., 2024b; Beigi et al., 2024b) or misinformation (Jiang et al., 2024b; Yang et al., 2023; Beigi et al., 2024a) and moderating the information exchange process.”

“In SMoA, we also borrow this insight and adopt role-playing (Wang et al., 2023; Li et al., 2023a), a widely adopted technique in personalized dialogue to improve the diversity among each processor.”

“In Figure 3a, we analyze alignment and fairness attributes using the Just-Eval and CEB datasets.”

“Scalability is a crucial factor in evaluating multi-agent LLM systems (Qian et al., 2024).”

“In all, SMoA performs well in identifying reasoning error (Tong et al., 2024; An et al., 2023) and filtering out unsafe content (Tan et al., 2024b; Beigi et al., 2024b) or misinformation (Jiang et al., 2024b; Yang et al., 2023; Beigi et al., 2024a) and moderating the information exchange process.”

“For reasoning, we use the Massive Multitask Agent Understanding (MMAU) benchmark (Yin et al., 2024), which involves challenging reasoning tasks including math understanding, tool-using and code contests.”

“Additionally, there are some works that propose mixture-of-agents, to guide various LLMs to collaborate with each other by referring to others’ responses and generating their own continuation (Zhang et al., 2023; Li et al., 2023b; Wang et al., 2024b).”

“In multi-agent collaboration systems, both Liang et al. (2023) and Li et al. (2024b) propose to leverage the judge LLM to select the most feasible and reasonable solutions among multiple candidates’ responses.”

“Similarly, Li et al. 156 utilized a judge LLM in layer-based multi-agent collaboration to improve response quality and efficiency.”

“• StreamBench evaluates agents in continuous improvement settings, testing their ability to adapt and improve over time with tasks like Text-to-SQL generation, Python programming. and medical diagnostics.”

“Drawing in-spiration from expert-guided problem-solving practices (Chen et al., 2025b; Li et al., 2024a), our framework decomposes the error detection work-flow into three synergistic agents (refer to Figure 1(c)): an image-text consistency validator to detect semantic consistency, a visual…”

“…studies have explored diverse debate strategies, including adversarial settings—where agents take opposing sides and a third agent acts as judge [Liang et al., 2024]— and collaborative approaches, where agents work together to iteratively solve a problem [Li et al., 2025a, Estornell et al., 2025].”

“However, these debate methods typically rely on single homogeneous LLM. Recently, mixture of agents (MOA) [16, 55] Another class of multi-agent collaboration approach is model routing [56, 57, 58, 59, 60], which trains router networks to distribute questions to specialized models.”

“…et al., 2025b; Tan et al., 2025), code generation (Roziere et al., 2023; Ma-jumdar et al., 2025), reinforcement learning (Liang et al., 2025a,b), data augmentation (Wang et al., 2024a; Liang et al., 2024), and retrieval-augmented generation (Xu et al., 2025a; Li et al., 2025a; Yang et al., 2025b).”

“MoA [23, 42] further proposes architectural hierarchies and network-based communication patterns to enhance collective reasoning.”

“…routing to one or a few experts [6, 35, 63, 39, 15, 20, 28, 26, 60, 58, 23], sometimes with ensemble sampling [58]; (ii) response-level collaboration (mixture-of-agents) [48, 31, 33, 4]; and (iii) parameter-space fusion (merging, continual learning, distillation) [51, 55, 14, 21, 59, 52, 47].”

“Subsequently, the mixture-of-agents (MoA) paradigm leveraging multi-agent collaboration is proposed to deal with more complex problems by combining the strengths of diverse LLMs (Du et al. 2023; Zhang et al. 2024c; Wang et al. 2024; Li et al. 2024a).”

“Sparse Mixture-of-Agents (SMoA) (D. Li et al., 2024) builds on MoA.”

“A prevalent strategy to enhance collective intelligence in these systems is collaborative expertise specialization, where LLMs are instructed to simulate specific expert personas (e.g., “act as an experienced lawyer”) (Li et al., 2024a; Xu et al., 2024a).”

“, driving substantial progress in automatic assessment (Beigi et al., 2024; Hu et al., 2024; Jeong et al., 2024; Li et al., 2025b; Tan et al., 2025).”

“…and error steps shifted toward lower values, confirming that our reconstruction– prototype framework captures the causal structure of multi-agent reasoning and enables robust error detection via simple thresholding, with complete results and additional visualizations provided in the appendix.”

“Multiagent Debate (Du et al., 2024), RECONCILE (Chen et al., 2024a), RR-MP (He et al., 2025), ChatEval (Chan et al., 2024) CoA (Zhang et al., 2024b), Corex (Sun et al., 2023), CoMM (Chen et al., 2024b), DoT (Shao et al., 2025), Instilling Parallel Reasoning (Macfarlane et al., 2025) Mixture of Agents MoA (Wang et al.,…”

“There are some methods that fall into both categories, such as mixture of agents (MoA) [7, 31, 58].”

“In a subsequent work [7], sparse mixture-of-agents (SMoA) framework was proposed to further improve the efficiency, diversity, and scalability of multi-agent LLMs by introducing sparsity in agent interactions through mechanisms like response selection and early stopping.”

“…further by learning to merge several complete outputs into a single consolidated response, boosting overall quality but still facing challenges in generalizing across diverse tasks and overlooking valuable token-level probability cues generated during decoding (Lin et al., 2025; Li et al., 2025).”

“In this work, we consider a system consisting of n users generating binary (true/false) queries, a central task processor, i.e., router, and a set of LLMs specialized to handle specific types of user queries, as illustrated in Fig.”

“Even with non-reasoning LLMs, there is a direct relation between more tokens and better output, as shown in the works, like mixture-of-agents [5], [6], where the responses from different LLMs were appended to get better response.”

“However, some recent studies also report cases where LLM-based self-improvement does not bring a significant boost and can even degrade the model’s performance (8; 9).”

“Recent work on sparse mixture-of-agents has also explored sparse information flow in multi-agent LLMs from the perspectives of response selection and early stopping mechanisms(Li et al. 2024a).”

“Previous work [68] has shown that LLMs exhibit varying levels of semantic understanding and reasoning ability depending on their pretraining.”

“Within information consolidation , beyond RMoA (Xie et al., 2025), SMoA (Li et al., 2025a) sparsifies agent interactions via dedicated judge and moderator agents, while Self-MoA (Li et al., 2025b) instead argues quality dominates diversity, aggregating samples from a single strong model.”

“Previous works [23], [66], [67] have demonstrated that leveraging the collective expertise of multiple LLMs can lead to the development of more capable and robust models for a variety of tasks.”

“Sparse-MoA significantly reduced token consumption, albeit with the overhead of maintaining additional control agents.”

“…2026); multi-agent frameworks: Deep Agents, CrewAI, and AutoGen (LangChain AI, 2026; CrewAI, Inc., 2026; Wu et al., 2024); fixed collaboration methods: Peer Review, SMoA, and OMAC (Xu et al., 2023; Li et al., 2025; 2026b); and automated MAS design: ADAS and EvoMAS (Hu et al., 2025; 2026b).”

“Early efforts such as Debate, Reflect, and Distill (DRD) distill multi-agent debate trajectories into a student model [54], while SMoA emulates MoA-style aggregation using a single LLM that generates multiple candidate responses [20].”

“We compare Direct LLM Call as a control; fixed organizations Peer Review (Xu et al., 2023) and SMoA (Li et al., 2025); automatic design methods ADAS (Hu et al., 2025) and OMAC (Li et al., 2026a); and the self-evolving baseline EvoMAS (Hu et al., 2026).”

“• SMoA Li et al. (2025): A general-purpose Sparsified Mixture of Agents framework that harnesses collaborative capabilities of multiple heterogeneous language models.”

“Li et al. [32] further demonstrate that integrating a judge LLM within a layered multi-agent collaboration pipeline can improve both response quality and computational efficiency.”

“To overcome this, the Sparse Mixture-of-Agents (SMoA) architecture keeps the computational burden in check by sparsifying the flow of information [26].”

“We then analyze SMoA [22] under two settings: with and without early-stopping across a layer depth of 3.”

“Recent efforts like SLM-Bench [19] integrate environmental metrics along-side accuracy across diverse NLP tasks, while multi-agent frameworks [20], [21] have been proposed for iterative evaluation enhancement.”