Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling

May combine two claims.

Multi-agent systems can outperform single-agent baselines but typically require more compute.

“Multi-agent systems can outperform single-agent baselines (Wang et al., 2024a;Du et al., 2023), but they typically require more compute.”

• cites Wang et al., 2024a sourceSupported
• cites Du et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims. The quote only partly states this claim.

The observation that MoA scales further than debate may be explained by its architecture: adding layers does not increase context length per model, while increasing debate rounds expands context and computational overhead.

“The observation that MoA scales further than debate may be explained by its architecture.”

• cites Wang et al., 2024a sourcePartly supported

  Section 2.2 describes MoA’s layered architecture and says it need not concatenate the prompt with all model responses. But the paper does not report that MoA scales further than debate or compare how added layers and debate rounds affect context length and computational overhead; its discussion of debate in Section 4.2 is background only. The proposed explanation is therefore not established by this source. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Du et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Huang et al. (2024) observed that debate does not outperform self-consistency under a similar compute budget.

“They use three agents and two rounds in debate as in the initial debate paper (Du et al., 2023) and observe that debate does not outperform self-consistency under a similar compute budget.”

• cites Huang et al., 2024 sourceSupported
• cites Du et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

To mitigate model-specific biases, the study adopts the single-proposer setting, where only a single LLM is used for all proposer models.

“To mitigate model-specific biases, we adopt the singleproposer setting (Wang et al., 2024a), where only a single LLM is used for all proposer models.”

• cites Wang et al., 2024a sourceSupported

May combine two claims.

Self-consistency saturates earlier than multi-agent methods, and multi-agent gains persist, particularly on more complicated tasks.

“While self-consistency saturates earlier, multi-agent gains persist, particularly on more complicated tasks.”

• Also stated in: § 4.1 Are multi-agent systems more computeefficient than single-agent methods? · plain

  “Both self-consistency and debate exhibit saturation at around 10 sequences and 4 agents, respectively, whereas MoA continues to improve nearly linearly up to 5 models and 4 layers.”

May combine two claims.

In debate, accuracy improves when scaling agents up to four, and decreases with more agents.

“Accuracy improves when scaling agents up to four, and decreases with more agents.”

May combine two claims.

On MMLU-Pro, MoA with 5 models and 4 layers achieves 71.4% accuracy, compared to the CoT baseline of 64.3%, a gain of +7.1 percentage points.

“On MMLU-Pro, MoA improves the 70B-model accuracy from 64.3% (CoT) to 71.4% (+7.1 pp), outperforming self-consistency (68.7%) and debate (70.0%) by +2.7 and +1.4 percentage points under comparable inference budgets.”

• Also stated in: § 4.1 Are multi-agent systems more computeefficient than single-agent methods? · plain

  “Across the tested range of up to ~20×CoT compute, the highest accuracies are 68.7% for self-consistency (10 iterations), 70.0% for debate (4 agents, 2 layers), and 71.4% for MoA (5 models, 4 layers).”

May combine two claims.

Self-refinement consistently yields lower accuracy than the CoT baseline despite requiring more compute.

“Self-refinement consistently yields lower accuracy than the CoT baseline despite requiring more compute.”

May combine two claims.

Multi-agent pipelines are more efficient than single-agent baselines when scaled Pareto-optimally, with MoA dominating the Pareto-front.

“Multi-agent pipelines are more efficient than singleagent baselines when scaled Pareto-optimally, with MoA dominating the Pareto-front.”

• Also stated in: § Key Contributions: · boosted

  “We demonstrate that multi-agent systems are more efficient than single-agent methods when scaled optimally.”

• Also stated in: § 4.1 Are multi-agent systems more computeefficient than single-agent methods? · plain

  “MoA configurations consistently lie closer to the Paretofront than self-consistency, thereby delivering higher compute-accuracy efficiency.”

May combine two claims.

Under comparable inference budgets on MMLU-Pro, debate and MoA outperform self-consistency by 1.3 and 2.7 percentage points, respectively.

“With an equal computing budget, debate and mixture-of-agents outperform self-consistency by 1.3% and 2.7% points, respectively.”

• Also stated in: § 4.1 Are multi-agent systems more computeefficient than single-agent methods? · plain

  “Relative to self-consistency under comparable compute budgets, debate and MoA gain +1.3 and +2.7 percentage points, solely by using a different scaling paradigm.”

May combine two claims.

In debate, scaling rounds degrades performance with two agents and has no noticeable effect with more agents.

“Scaling rounds degrades performance with two agents and has no noticeable effect with more agents.”

May combine two claims.

MoA configurations consistently lie closer to the Pareto-front than self-consistency, delivering higher compute-accuracy efficiency.

“MoA configurations consistently lie closer to the Paretofront than self-consistency, thereby delivering higher compute-accuracy efficiency.”

May combine two claims.

In MoA, accuracy improves as models or layers are scaled toward the point where the number of models exceeds the number of layers by one, with Pareto-optimal performance at 3 models/2 layers, 4 models/3 layers, and 5 models/4 layers.

“Accuracy improves as models or layers are scaled toward the point where the number of models exceeds the number of layers by one, with Pareto-optimal performance observed at this specific ratio (3 models with 2 layers, 4 models with 3 layers, and 5 models with 4 layers).”

May combine two claims.

On MMLU-Pro, medium-difficulty tasks have 53.0% CoT accuracy and improve by 8.5 percentage points when compute is increased to 15-20 times the CoT compute.

“Medium-difficulty tasks start with a CoT accuracy of 53.0%, which improves by 8.5% points”

May combine two claims.

On MMLU-Pro, hard tasks have 8.4% CoT accuracy and gain 9.0 percentage points as test-time compute increases.

“hard tasks have an 8.4% CoT accuracy, which gains 9.0% points as test-time compute increases.”

May combine two claims.

On MMLU-Pro, easy tasks have 94.4% CoT accuracy and improve by 2.2 percentage points when compute is increased to 15-20 times the CoT compute.

“Easy tasks have a 94.4% CoT accuracy, which improves by 2.2% points when compute is increased to an equivalent of 15-20 times the CoT compute.”

May combine two claims.

Larger models not only deliver higher performance but can also be more compute-efficient than heavily scaled smaller models.

“Larger models not only deliver higher performance but can also be more compute-efficient than heavily scaled smaller models.”

May combine two claims.

Under comparable compute budgets, debate yielded a 1.3% increase in accuracy and MoA a 2.7% increase over self-consistency, driven solely by their different scaling paradigms.

“Under comparable compute budgets, debate yielded a 1.3% increase in accuracy, and MoA a 2.7% increase over self-consistency, driven solely by their different scaling paradigms.”

May combine two claims.

Across the pipelines self-consistency, self-refinement, debate, and MoA, multi-agent systems consistently outperform single-agent approaches, with MoA dominating the Pareto front across benchmarks.

“Across the pipelines self-consistency, selfrefinement, debate, and MoA, we found that multiagent systems consistently outperform single-agent approaches, with MoA dominating the Paretofront across benchmarks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For measured accuracies between 0.63 and 0.72, the 95% binomial confidence interval uncertainty ranges from 0.028 to 0.03.

“For measured accuracies between 0.63 and 0.72, the uncertainty ranges from 0.028 to 0.03.”

May combine two claims.

Using a single small 8B proposer has little impact on accuracy but yields only minimal compute savings because it accounts for only a small fraction of proposer generations.

“While using a single small 8B proposer has little impact on accuracy, it yields only minimal compute savings because it accounts for only a small fraction of proposer generations.”

May combine two claims.

As the number of smaller proposer models grows, performance declines more substantially, reaching the lowest accuracy when all proposers have 8B parameters.

“As the number of smaller proposer models grows, performance declines more substantially, reaching the lowest accuracy when all proposers have 8B parameters.”

May combine two claims.

Additional test-time compute is most beneficial for harder tasks, whereas easy tasks exhibit diminishing returns.

“In addition, our analysis across task difficulty levels demonstrated that additional testtime compute is most beneficial for harder tasks, whereas easy tasks exhibit diminishing returns.”

May combine two claims.

The MoA paradigm increased accuracy by 7.1 percentage points at ~20× the CoT budget; if compute is tight, a ~5× CoT budget already captures most of that gain.

“This paradigm increased accuracy by 7.1% points at ~20× the CoT budget; but if compute is tight, a ~5×CoT budget already captures most of that gain.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because multi-agent systems are computationally expensive to run, the study randomly shuffles tasks and draws a sample of 1000 questions.

“Since multi-agent systems are computationally expensive to run, we randomly shuffle tasks and draw a sample of 1000 questions.”

May combine two claims.

Sequential scaling is less detrimental in MoA than in debate or self-refinement, likely because context is not accumulated across layers.

“Sequential scaling is less detrimental than in debate or selfrefinement, likely because the context is not accumulated across layers.”

May combine two claims.

Both parallel and sequential scaling are beneficial in the MoA system.

“This result shows that both parallel and sequential scaling are beneficial in the MoA system.”

Classified as probably not a substantial claim.

To ensure fair comparisons, the study keeps as many parameters constant as possible and varies only those relevant to each experiment.

“To ensure fair comparisons, we keep as many parameters constant as possible and vary only those relevant to each experiment.”

Classified as probably not a substantial claim.

The study uses models of the same family to separate the effects of inference-time scaling from those of model capacity.

“We use models of the same family to separate the effects of inference-time scaling from those of model capacity.”

Classified as probably not a substantial claim. May combine two claims.

Compute costs were estimated using theoretical formulas that account for FLOPs and memory transfer, providing a consistent basis for comparison, but real-world runtime may vary due to framework overhead, batching strategies, or system-level constraints.

“Our compute costs were estimated using theoretical formulas that account for FLOPs and memory transfer.This provides a consistent basis for comparison; however, real-world runtime may vary due to framework overhead, batching strategies, or system-level constraints.”

• AssumptionRests on an assumption: The theoretical compute cost model based on FLOPs and memory transfer is a valid basis for comparing compute efficiency across configurations.

Classified as probably not a substantial claim.

For each stage, compute time and memory transfer time are estimated following the formulation of Li, Linden (2023).

“For each stage, we estimate both the compute time and the memory transfer time, following the formulation of Li, Linden (2023), with the specific equations summarized in Table 4.”

• cites Li, Linden (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Table 4Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Since computation and memory transfer occur in parallel, the slower of the two determines the effective runtime.

“Since computation and memory transfer occur in parallel, the slower of the two determines the effective runtime.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

To assess robustness, the experiments from Figure 2 were rerun with one differing factor: replacing MMLU-Pro with BBH, and using a smaller 8B-parameter model instead of the 70B-parameter model.

“To assess the robustness of our results, we reran the experiments from Figure 2 with one differing factor.In Figure 6, we replaced the MMLU-Pro benchmark with the BBH benchmark and in Figure 7, we used a smaller 8B-parameter model of the same model family instead of the 70B-parameter model.”

• cites Figure 2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Figure 6Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Figure 7Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

While individual configurations may have overlapping 95% confidence intervals, clear patterns emerge when examining the results as a whole, as patterns persist for similar configurations.

“While individual configurations may have overlapping 95% confidence intervals, clear patterns emerge when examining the results as a whole, as patterns persist for similar configurations (e.g., other benchmarks, model size, near-the-same pipeline parameters).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study was limited to non-reasoning models.

“Finally, our study was limited to non-reasoning models.”

Classified as probably not a substantial claim.

The study controlled for compute in terms of runtime that accounts for both FLOP and memory transfer times.

“We controlled for compute in terms of runtime that accounts for both FLOP and memory transfer times.”

May combine two claims.

Most studies focus on FLOPs as the primary measure of compute and neglect memory transfer, which also markedly influences runtime.

“While most studies focus on FLOPs as the primary measure of compute, they neglect memory transfer, which also markedly influences runtime (Li, Linden, 2023).”

• cites Li, Linden, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Generation can be divided into a prefill stage, in which the model processes all input tokens once to produce and cache key-value states, and a decode stage, in which the model generates output tokens one by one, reusing the cached states.

“In addition to these factors, generation can be divided into two stages: a prefill stage, in which the model processes all input tokens once to produce and cache key-value states, and a decode stage, in which the model generates output tokens one by one, reusing the cached states (Li, Linden, 2023).”

• cites Li, Linden, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In deployment scenarios with limited GPU memory, the efficiency gains of larger models may not be realizable.

“In deployment scenarios with limited GPU memory, such efficiency gains may not be realizable, making it important to explore trade-offs under constrained hardware conditions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study used a sample size of n=1000 instead of the full benchmark, which introduces some variance.

“Since multi-agent systems are computationally expensive to run, we used a sample size of n=1000 instead of the full benchmark.This introduces some variance.”