CoChat
Sign Up
PreprintCited by 26Topic ModelingMetaheuristic Optimization Algorithms ResearchNatural Language Processing Techniques

Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

Open access
Feng, Shangbin, Wang, Zifeng, Goyal, Palash, Wang, Yike, Shi, Weijia, Xia, Huang, Palangi, Hamid, Zettlemoyer, Luke +3 more·2025·arXiv
DOI 10.48550/arxiv.2502.04510arXiv 2502.04510
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models.

AI snapshot

Written from the paper’s full text.

TL;DR

HETEROGENEOUS SWARMS jointly optimizes model roles and weights for multi-LLM systems by representing them as DAGs and alternating particle-swarm optimization over graph structures and model weights. It reports gains over 17 role- and/or weight-based baselines across 12 tasks.

Key findings

  • HETEROGENEOUS SWARMS achieves the best performance on 11 of 12 datasets and outperforms the second-best approach by 18.5% on average across tasks.
  • It achieves consistent positive collaborative gains with an average of 0.213; in the bucket where none of the initial LLMs could solve a problem individually, it solves 18.1% of them on average.
  • Scaling the collaboration from 2 to 10 LLMs gives a 27.1% improvement on average, and increasing initial LLM diversity from least to most diverse gives an 89% average relative improvement.
  • Role and weight importance varies by task: for knowledge tasks, weight baselines outperform role baselines by 4.3% on average, while for agent tasks role baselines are 9.2% better; dynamic weight outperforms static weight by 30.1%, and dynamic role outperforms static role by up to 8.2%.

Methods

  • The method represents a multi-LLM system as a directed acyclic graph of LLMs with continuous adjacency matrices. The role-step decodes matrices into discrete DAGs with G-decode, calls LLMs in topological order, evaluates a utility function, and optimizes adjacency matrices with particle swarm optimization. The weight-step uses JFK-score to quantify each LLM’s individual contribution across random assignments in the best-found DAG, then optimizes model weights with PSO.
  • Experiments use a pool of 10 LLM experts from Feng et al., fine-tuned from GEMMA-7B on 10 Tulu-v2 domains, with MISTRAL-7B also tested. Evaluation covers 12 datasets across knowledge, reasoning, agent, and miscellaneous categories, compared against 17 baselines.
  • Main hyperparameters include top-p = 0.8, N = 10, M = 10, search patience 6, max iteration 20, with grid search over other hyperparameters; experiments run on a cluster with 16 A100 40GB GPUs. Evaluation uses multiple-choice, exact match, and Gemini-as-a-judge, with 200 examples for optimization and 1,000 for evaluation by default.

Limitations

  • The authors state that HETEROGENEOUS SWARMS by default requires a pool of LLMs sharing the same architecture for weight-step optimization; with different architectures, it could use only the role-step or token-level collaboration for the weight step.
  • The authors state that both optimization and inference call multiple LLMs, increasing potential cost; they propose sparsity, dropout, and time-complexity analysis as mitigations.
  • The authors state they did not evaluate black-box models in the LLM pool, tools/APIs as nodes, cycles in the multi-LLM network, or instance-level changes with a learnable edit function.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.04510

Cites

128 of 128 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    HETEROGENEOUS SWARMS outperforms 17 role- and/or weight-based baselines by 18.5% on average across 12 tasks.

    “Experiments demonstrate that HETEROGE-NEOUS SWARMS outperforms 17 role-and/or weight-based baselines by 18.5% on average across 12 tasks.”
    • Also stated in: § 5 Results · plain
      “HET-EROGENEOUS SWARMS achieves the best performance on 11 of the 12 datasets, outperforming the second-best approach by 18.5% on average.”
    • Also stated in: § Models and Implementation · plain
      “HETEROGE-NEOUS SWARMS outperforms TRIVIAL, STATIC WEIGHT, DYNAMIC WEIGHT, STATIC ROLE, and DYNAMIC ROLE approaches by 18.5% on average across tasks.”
    Quote in the paperSubstantial claim
  • Findingown data

    HETEROGENEOUS SWARMS achieves the best performance on 11 of the 12 datasets.

    “HET-EROGENEOUS SWARMS achieves the best performance on 11 of the 12 datasets, outperforming the second-best approach by 18.5% on average.”
    Quote in the paperSubstantial claim
  • Findingown data

    The diversity of initial LLMs does help greatly, with an average relative improvement of 89% from the least to most diverse.

    “Figure 5 illustrates that the diversity of initial LLMs does help greatly, with an average relative improvement of 89% from the least to most diverse.”
    Quote in the paperSubstantial claim
  • Findingnone given

    If the network is dense, i.e. when one LLM takes the output of too many LLMs as input, it might lead to irrelevant outputs.

    “We observe that if the network is dense, i.e. when one LLM takes the output of too many LLMs as input, it might lead to irrelevant outputs.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Scaling the number of LLM experts from 2 to 10 results in consistent improvements across 4 datasets.

    “Scaling the number of LLM experts from 2 to 10 results in consistent improvements across 4 datasets.”
    Quote in the paperSubstantial claim
  • May combine two claims. 16
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 5

  • May combine two claims.

    Findingcitation

    HETEROGENEOUS SWARMS enables inference-time scaling of smaller language models through topological collaborative generation, with a 27.1% improvement on average when scaling the collaboration from two to ten LLMs.

    “HETEROGENEOUS SWARMS also enables inference-time scaling [82,5,100] of smaller language models through topological collaborative generation, with a 27.1% improvement on average when scaling the collaboration from two to ten LLMs.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [82]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [100]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • The quote only partly states this claim.

    Findingown data

    Randomly skipping the role-step or weight-step with dropout rates dr or dw in {0.2,0.5,0.8} causes only a minor performance drop.

    “It is illustrated that the acceleration only comes with a minor performance drop, showing the flexibility of HETEROGENEOUS SWARMS to adapt to different computational budget.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    L1 normalization better retains performance while thresholding presents a larger reduction in inference cost.

    “Results demonstrate that L 1 better retains performance while thresholding presents a larger reduction in inference cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Individual LLMs have heterogeneous role distributions given their topological position: for branching nodes there is often higher divide, while for converging nodes there is often higher refine and feedback.

    “We find that individual LLMs do have heterogeneous role distributions given their topological position: for the branching nodes there is often higher divide, while for the converging nodes there is often higher refine and feedback.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Irrelevant roles gradually decrease while the three other functional roles consistently increase in the course of optimization.

    “We observe that irrelevant roles gradually decrease while the three other functional roles consistently increase in the course of optimization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the graph reasoning task NLGraph, there is greater divide and conquer to solve part of the problem; for the knowledge tasks such as K-Crosswords, there is greater feedback to identify knowledge gaps in existing answers.

    “For the graph reasoning task NLGraph, there is greater divide and conquer to solve part of the problem; for the knowledge tasks such as K-Crosswords, there is greater feedback to identify knowledge gaps in existing answers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    HETEROGENEOUS SWARMS achieves collaborative gains (C-Gain) of 0.143, 0.184, 0.101, and 0.426 on the four datasets, all greater than 0.

    “HETEROGENEOUS SWARMS achieves collaborative gains (C-Gain) of 0.143, 0.184, 0.101, and 0.426 on the four datasets, all > 0, and demonstrate consistent collaborative gains.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    If the pool of LLMs contain different architectures, then HETEROGENEOUS SWARMS could either operate with only the role-step to optimize the multi-LLM network, or employ token-level collaboration for the weight step.

    “If the pool of LLMs contain different architectures, then HETEROGENEOUS SWARMS could either operate with only the role-step to optimize the multi-LLM network, or employ token-level collaboration for the weight step similar to Liu et al. [58] and Feng et al. [26].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Liu et al. [58]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Feng et al. [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    PSO uniquely enables leveraging multiple diverse collaboration patterns and specialized model checkpoints, composing their expertise and strengths in multi-LLM systems.

    “We employ PSO for the optimization, since it uniquely enables us to leverage multiple diverse collaboration patterns and specialized model checkpoints, composing their expertise and strengths in multi-LLM systens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study used open-access and publicly available data.

    “We used open-access and publicly available data (Section D)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study employs a one-tailed z-test and reports statistical significance test results on the datasets.

    “We also employ the z-test with the one-tailed hypothesis and present statistical significance test results on the datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    At optimization time, the role step calls model inference O(nN) times and the weight step O(nM) times.

    “At optimization time, the role step calls model inference O(nN ) times and the weight step O(nM ) times”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The number of model inferences is the main computational cost.

  • May combine two claims.

    Interpretationown data

    HETEROGENEOUS SWARMS could speed up with one or multiple strategies used in conjunction.

    “we show that HETEROGENEOUS SWARMS could speed up with one or multiple strategies used in conjunction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Encouraging sparsity in the network reduces context lengths and hence inference costs.

    “In addition, encouraging sparsity in the network would also reduce context lengths and hence inference costs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    HETEROGENEOUS SWARMS uniquely offers a flexible methodology to jointly optimize the roles and weights of diverse LLMs, discovering and adapting novel multi-LLM systems.

    “HETEROGENEOUS SWARMS uniquely offers a flexible methodology to jointly optimize the roles and weights of diverse LLMs, discovering and adapting novel multi-LLM systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    New tasks and domains would require substantial prompt engineering, which heavily depends on prior knowledge of the given task.

    “However, new tasks and domains would require substantial prompt engineering, which heavily depends on prior knowledge of the given task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Fixed-role systems usually orchestrate LLMs in a fixed workflow and seed LLMs with different hand-crafted prompts and enable their interaction of message passing based on these prompt-induced roles.

    “Fixed-role systems usually orchestrate LLMs in a fixed workflow and seed LLMs with different hand-crafted prompts and enable their interaction of message passing based on these prompt-induced roles [80,25].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [80]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    HETEROGENEOUS SWARMS could find adapted multi-LLM systems with new compositional skills and substantial collaborative gains.

    “This indicates that HETEROGENEOUS SWARMS could find adapted multi-LLM systems with new compositional skills and substantial collaborative gains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Both the optimization and inference process of HETEROGENEOUS SWARMS features calling multiple LLMs, increasing its potential cost.

    “Both the optimization and inference process of HETEROGENEOUS SWARMS features calling multiple LLMs, increasing its potential cost.”
    Quote in the paperNot a substantial claim
19 of 19

  • The authors state this as an assumption.

    Multiple large language models complement each other based on their diverse expertise.

    “HETEROGENEOUS SWARMS operates on the assumption that the multiple large language models would complement each other based on their diverse expertise.”

  • The paper relies on this without support.

    The utility function f is a valid proxy for multi-LLM system performance.

    “we run grid search over other hyperparameters and report the best-found expert based on utility function f .”
    “We run up to 50 to 200 runs by randomly choosing over these hyperparameter search settings and report the best-found expert on utility function f .”

  • The paper relies on this without support.

    Gemini-as-a-judge ratings are a valid measure of response quality.

    “Table 10: Default Gemini-as-a-judge evaluation prompt.”

  • The paper relies on this without support.

    The number of model inferences is the main computational cost.

    “We measure the complexity by the number of model inferences as they are the main computational cost.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • G-Decode is assigned the wrong algorithm number

    Minor

    The figure caption calls G-Decode Algorithm 2, but the paper labels Algorithm 2 as JFK-score and Algorithm 4 as the DAG decoder.

  • JFK-score formula uses one assignment's utility for every term

    Minor

    In the stated formula, f(X_i) does not vary with the summation index j, so the weighted average reduces to f(X_i). The worked example instead weights different assignments' utilities, as the description of JFK-score requires.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. Acc-debate: An actor- critic approach to multi-agent debate. In The Thirteenth International Conference on Learning Representations, 2025.
  2. Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael Lyu. Competing large language models in multi-agent gaming environments. In The Thirteenth International Conference on Learning Representations, 2025.
  3. Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. Breaking mental set to improve reasoning through diverse multi-agent debate. In The Thirteenth International Conference on Learning Representations, 2025.
  4. Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled architecture for autonomous llm agents. In Forty-first International Conference on Machine Learning, 2024.
  5. Fan Yao, Yuwei Cheng, Ermin Wei, and Haifeng Xu. Single-agent poisoning attacks suf- fice to ruin multi-agent learning. In The Thirteenth International Conference on Learning Representations, 2025.
  6. Xinjie Yao, Yu Wang, Pengfei Zhu, Wanyu Lin, Jialu Li, Weihao Li, and Qinghua Hu. So- cialized learning: making each other better through multi-agent collaboration. In Forty-first International Conference on Machine Learning, 2024.
  7. Hangfan Zhang, Zhiyao Cui, Qiaosheng Zhang, and Shuyue Hu. Multi-llm-agents debate- performance, efficiency, and scaling challenges. In The Fourth Blogpost Track at ICLR 2025, 2025.
  8. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, 2024.

Cited by

Sorted by citations

Related

Authors
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., & Pfister, T.
Affiliations
  • University of Washington (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
126786-126818
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2502.04510
arXiv
2502.04510
OpenAlex
W4407308863
Semantic Scholar
b50e52593b5b426271be78faa1f09607d1a1769a
Cited by
0
References
120
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • U.S. Department of Defense (HR00112520300)
  • Defense Advanced Research Projects Agency (HR00112520300, FA8650-23-C-7316)
  • Advanced Research Projects Agency
  • Air Force Research Laboratory
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Metaheuristic Optimization Algorithms Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
model weight optimization, directed acyclic graphs, particle swarm optimization, swarm intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review