CoChat
Sign Up
PreprintCited by 5Multi-Agent Systems and NegotiationAI-based Problem Solving and PlanningNatural Language Processing Techniques

Towards a Science of Scaling Agent Systems

Open access
Kim, Yubin, Gu, Ken, Park, Chanwoo, Park, Chunjong, Schmidgall, Samuel, Heydari, A. Ali, Yan, Yao, Zhang, Zhihan +12 more·2025·arXiv
DOI 10.48550/arxiv.2512.08296arXiv 2512.08296
Open this paper in CoChat — free account

Abstract

Agents, language model-based systems capable of reasoning, planning, and acting are widely adopted in real-world tasks, yet how their performance changes as these systems scale across key dimensions remains underexplored. We introduce quantitative scaling principles for agent systems as a predictive model, capturing how performance varies with coordination, model capability, and measurable system and task factors. Across 260 configurations spanning six agentic benchmarks, five canonical architectures (Single-Agent and four Multi-Agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families, we perform controlled evaluations, standardizing tools, prompts, and compute to isolate architectural effects. The resulting model achieves a cross-validated R^2=0.373 across all six benchmarks (R^2=0.413 with a task-grounded capability metric). We identify a robust capability-saturation effect and additional patterns: (1) a coordination yields diminishing returns once single-agent baselines exceed certain performance; (2) tool-heavy tasks appear to incur multi-agent overhead; and (3) architectures without centralized verification tend to propagate errors more than those with centralized coordination. Relative performance change compared to single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning, demonstrating that architecture-task alignment determines collaborative success. The framework identifies the best-performing architecture for 87% of held-out configurations and shows consistent relative architecture preferences on unseen frontier models. Agent effectiveness depends on alignment between coordination and task structure, and that mismatched coordination degrades the performance.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper presents a controlled, quantitative scaling study of LLM-based agent systems across 260 configurations, six agentic benchmarks, five architectures, and three model families. It finds that multi-agent coordination is not universally beneficial: architecture-task alignment, capability saturation, tool complexity, and coordination overhead determine whether MAS improves or degrades performance.

Key findings

  • Multi-agent performance is highly task-dependent, ranging from +80.8% over single-agent on Finance Agent under centralized coordination to −70.0% on PlanCraft under independent coordination.
  • A capability-saturation effect is the most robust finding: once single-agent baselines exceed roughly 45% accuracy, additional agents tend to yield diminishing or negative returns; this survives cluster-robust inference (p = 0.004) and Holm-Bonferroni correction (p_Holm = 0.018).
  • Tool-heavy tasks incur a coordination tax: the efficiency × tool-count interaction is β = −0.096, p = 0.002, with multi-agent efficiency penalties of 2–6× relative to single-agent systems.
  • Error amplification is architecture-dependent: trace-level error amplification is 17.2× for Independent MAS, 7.8× for Decentralized, 5.1× for Hybrid, and 4.4× for Centralized, versus 1.0× for SAS.
  • The regression model achieves cross-validated R² = 0.373 across all six benchmarks (R² = 0.413 with the task-grounded Agentic Capability Index) and selects the best-performing architecture for 87% of held-out configurations.

Methods

  • Controlled evaluation of 260 configurations across six agentic benchmarks: BrowseComp-Plus, Finance-Agent, PlanCraft, Workbench, SWE-bench Verified, and Terminal-Bench.
  • Five architectures: Single-Agent System (SAS) and four Multi-Agent variants—Independent, Centralized, Decentralized, Hybrid—instantiated across OpenAI, Google, and Anthropic model families, with tools, prompts, and total reasoning-token budgets matched across conditions.
  • Main analysis is a regression model predicting performance from model capability, agent count, tool count, single-agent baseline, and empirical coordination metrics (efficiency, overhead, trace-level error amplification, message density, redundancy), validated with 5-fold cross-validation, cluster-robust inference, Holm-Bonferroni correction, leave-one-dataset-out validation, and held-out frontier-model evaluation.

Limitations

  • The paper states that it compares canonical coordination structures with only preliminary scaling up to nine agents; all agents share identical base architectures differing mainly in scale and role prompts, and heterogeneous model mixing is only preliminarily explored.
  • It states that prompts were controlled but not optimized per model or family, that six benchmarks may not cover all agentic task types, and that SWE-bench Verified and Terminal-Bench use 20-instance subsets, yielding bootstrap 95% CIs typically around ±20 percentage points per cell.
  • It states that the regression clusters at the dataset level (G = 6), making cluster-robust standard errors conservative; several predictors significant under naive OLS lose significance under cluster-robust inference, and leave-one-dataset-out validation shows limited absolute cross-domain prediction.
  • Evident from the text: the Intelligence Index is not directly comparable across vendors despite identical scores, and the model systematically over-predicts Hybrid performance on held-out frontier models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2512.08296

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 12 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1. Introduction · p. 2

    Multi-agent systems impose intrinsic information fragmentation.

    “In contrast, multi-agent systems impose intrinsic information fragmentation [13]:”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [13] [13] Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of LLMs. arXiv preprint arXiv:2501.06322 , 2025. - source

      Section 4.4.2 (page unavailable) says decentralized agents operate on local information and may have limited communication, supporting the idea that information can be distributed among agents. But the paper does not establish information fragmentation as intrinsic to all multi-agent systems; its statement is specifically about decentralized systems. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1. Introduction · p. 2

    Single-agent systems maximize context integration by maintaining a unified memory stream in which all reasoning steps share full access to prior history, enabling effectively constant-time access to global context.

    “Single-agent systems maximize context integration by maintaining a unified memory stream in which all reasoning steps share full access to prior history , enabling effectively constant-time access to global context.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [27] [27] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning , 2023. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [28] [28] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=VtmBAGCN7o . - source

      Section 3.2 (p. 6) describes a shared message pool and selective subscriptions for MetaGPT’s multi-agent system, not a unified memory stream in a single-agent system. Searches for single-agent memory, full access to prior history, and constant-time access found no passage addressing those claims. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1. Introduction · p. 2

    Multi-agent systems incur an unavoidable coordination tax in which the global context must be compressed into inter-agent messages.

    “while parallel agents enable diverse exploration, they incur an unavoidable coordination tax in which the global context must be compressed into inter-agent messages.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [13] [13] Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of LLMs. arXiv preprint arXiv:2501.06322 , 2025. - source

      Section 4.2.1 (page unavailable) says frequent inter-agent communication can increase computational cost and complexity, supporting a coordination cost. Section 4.4.2 discusses agents with diverse expertise. The paper does not say this cost is unavoidable or that global context must be compressed into inter-agent messages. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Findingown data
    § Towards a Science of Scaling Agent Systems · p. 1

    Relative performance change compared to the single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning.

    “Relative performance change compared to single-agent baseline ranges from + 80 . 8% on decomposable financial reasoning to -70 . 0% on sequential planning”
    Quote in the paperSubstantial claim
  • Findingown data
    § Towards a Science of Scaling Agent Systems · p. 1

    Tasks where single-agent performance already exceeds 45% accuracy experience negative returns from additional agents, i.e., coordination yields diminishing returns once single-agent baselines exceed certain performance.

    “(1) a coordination yields diminishing returns once single-agent baselines exceed certain performance;”
    • Also stated in: § 1. Introduction · p. 3 · plain
      “tasks where single-agent performance already exceeds 45% accuracy experience negative returns from additional agents”
    Quote in the paperSubstantial claim
  • May combine two claims. 91
  • The quote only partly states this claim. 4
  • Classified as probably not a substantial claim. 25

  • May combine two claims.

    Premisecitation
    § 2. Related Work · p. 5

    On non-agentic benchmarks, multi-agent systems show monotonic improvement through ensemble effects, reaching 89% on HumanEval with five agents, as voting corrects errors without sequential compounding.

    “On non-agentic benchmarks, multi-agent systems show monotonic improvement through ensemble effects (89% on HumanEval with five agents), as voting corrects errors without sequential compounding [23].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [23] [23] Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. Transactions on Machine Learning Research , 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Zy4uFzMviZ . - source

      Section 2.1 (p. 2) says repeated model calls with majority voting can improve accuracy on several benchmarks, and that retrying can improve HumanEval results. It does not report monotonic gains from multi-agent voting, 89% accuracy with five agents, or the absence of sequential error compounding. Figure 1 (p. 4) describes five evaluation runs, not five agents. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 2. Related Work · p. 5

    In agentic settings, agents operate on divergent world states, with 34% overlap after 10 interactions.

    “agents operate on divergent world states (34% overlap after 10 interactions)”
    • Also stated in: § 1. Introduction · p. 2 · plain
      “agents operate on progressively divergent world states”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [23] [23] Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. Transactions on Machine Learning Research , 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Zy4uFzMviZ . - source

      Section 1.1 (p. 2) discusses agents acting in complex environments, but does not describe divergent world states. Searches for “34%,” “10 interactions,” and “world state” found no passage reporting the claimed overlap; the copy does not substantiate that measurement. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § Towards a Science of Scaling Agent Systems · p. 1

    Tool-heavy tasks incur multi-agent overhead, with efficiency penalties compounding as environmental complexity increases.

    “(2) tool-heavy tasks appear to incur multi-agent overhead;”
    • Also stated in: § 1. Introduction · p. 3 · plain
      “tool-heavy tasks (e.g., 16-tool business workflows) suffer from multi-agent coordination overhead, with efficiency penalties compounding as environmental complexity increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Towards a Science of Scaling Agent Systems · p. 1

    The resulting model achieves a cross-validated R2 = 0.373 across all six benchmarks, and R2 = 0.413 with a task-grounded capability metric.

    “The resulting model achieves a cross-validated 𝑅 2 = 0 . 373 across all six benchmarks ( 𝑅 2 = 0 . 413 with a task-grounded capability metric).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 14

    On SWE-bench Verified, all MAS architectures show slight degradation relative to SAS (mean 0.522): Hybrid -2.1% (0.511), Centralized -3.1% (0.506), Decentralized -5.4% (0.494), and Independent -14.9% (0.444).

    “On SWE-bench Verified, all MAS architectures show slight degradation relative to SAS (mean 0.522): Hybrid -2 . 1% (0.511), Centralized -3 . 1% (0.506), Decentralized -5 . 4% (0.494), and Independent -14 . 9% (0.444).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1. Setup · p. 11

    On BrowseComp-Plus, independent agents underperform SAS by 35%, while more structured coordination achieves modest gains.

    “(a) BrowseComp-Plus shows polarized results, with independent agents catastrophically underperforming relative to SAS (-35%) while more structured coordination achieves modest gains.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 4.2. Main Results · p. 14

    Domain complexity scores: Workbench 0.000, SWE-bench Verified 0.255, Finance Agent 0.407, Terminal-Bench 0.414, PlanCraft 0.419, BrowseComp-Plus 0.839.

    “We characterize each benchmark by a domain complexity score 𝐷 ∈ [ 0 , 1 ] (Appendix C), capturing the degree of sequential interdependence and empirical difficulty: Workbench (0.000, minimal sequential constraints) shows positive MAS returns or minimal overhead, SWE-bench Verified (0.255, decomposable engineering tasks) has low domain complexity but high single-agent baselines that trigger”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 13

    In PlanCraft, centralized multi-agent systems decompose inherently sequential tasks into artificial subtasks, generating substantial coordination messages and consuming token budget on coordination rather than reasoning.

    “In contrast, centralized multi-agent systems decompose inherently sequential tasks into artificial subtasks:”
    • Also stated in: § 4.2. Main Results · p. 13 · plain
      “This unnecessary decomposition generates substantial coordination messages on average for tasks requiring only a few execution steps, consuming token budget on coordination rather than reasoning.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.1. System Definition · p. 8

    Both error metrics consistently show that architectures with verification mechanisms contain errors more effectively than independent coordination, though they differ in absolute magnitude (Atask e ≈ 1.1-1.3 vs. Atrace e ≈ 4-17).

    “Both metrics consistently show that architectures with verification mechanisms contain errors more effectively than independent coordination, though they differ in absolute magnitude ( 𝐴 task 𝑒 ≈ 1 . 1-1 . 3 vs. 𝐴 trace 𝑒 ≈ 4-17)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1. Setup · p. 12

    Coordination overhead values relative to single-agent baseline are 0% for SAS, 58% for Independent, 263% for Decentralized, 285% for Centralized, and 515% for Hybrid.

    “Coordination complexity is parameterized by communication overhead: the total number of inter-agent message exchanges required per task, yielding empirical values ranging from 0% (SAS) to 515% (Hybrid), with Independent at 58%, Decentralized at 263%, and Centralized at 285% relative to the single-agent baseline (see Table 5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1. Introduction · p. 3

    The predictive framework using a mixed-effects regression model with empirical coordination metrics achieves cross-validated R2 = 0.373 across all six benchmarks, and R2 = 0.413 with a task-grounded capability metric, without dataset-specific parameters.

    “We establish a predictive framework using a mixed-effects regression model with empirical coordination metrics: efficiency (success/overhead ratio), error amplification factors, message density and redundancy as predictors, achieving cross-validated 𝑅 2 = 0 . 373 across all six benchmarks ( 𝑅 2 = 0 . 413 with a task-grounded capability metric), without dataset-specific parameters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1. Setup · p. 10

    Workbench (CV=0.12), Finance Agent (CV=0.18), and PlanCraft (CV=0.21) show lower variability, indicating more stable performance across configurations.

    “By comparison, Workbench (CV=0.12), Finance Agent (CV=0.18), and PlanCraft (CV=0.21) show lower variability, indicating more stable performance across configurations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1. Introduction · p. 4

    Decentralized coordination benefits tasks requiring parallel exploration of high-entropy search spaces, with dynamic web navigation showing +9.2%.

    “decentralized coordination benefits tasks requiring parallel exploration of highentropy search spaces (dynamic web navigation: + 9 . 2%)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 15

    Architecture effectiveness depends critically on domain structure: Finance Agent benefits most from Centralized (+80.8%) and Decentralized (+74.5%), Workbench from MAS-Decentralized (+5.6%), and BrowseComp-Plus from MAS-Decentralized (+9.2%).

    “Architecture effectiveness depends critically on domain structure: Finance Agent benefits most from Centralized (+80.8%) and Decentralized (+74.5%), Workbench from MAS-Decentralized (+5.6%), and BrowseComp-Plus from MAS-Decentralized (+9.2%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 1. Introduction · p. 4

    Higher capability, as measured by Intelligence Index, yields consistent linear returns.

    “while higher capability (Intelligence Index) yields consistent linear returns”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 1. Introduction · p. 4

    All multi-agent variants universally degrade performance on tasks requiring sequential constraint satisfaction, with planning showing -39% to -70%.

    “while all multi-agent variants universally degrade performance on tasks requiring sequential constraint satisfaction (planning: -39% to -70%)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 14

    Aggregating across all six benchmarks and architectures, overall mean MAS improvement is -0.3% (95% CI: [-58.7%, +77.2%]), with high variance (σ=37.5%).

    “Aggregating across all six benchmarks and architectures, the overall mean MAS improvement is -0 . 3% (95% CI: [ -58 . 7%, + 77 . 2%]), reflecting substantial performance heterogeneity with high variance ( 𝜎 = 37 . 5%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.1. Setup · p. 11

    On Terminal-Bench, Independent achieves marginal gains (+2%) while Centralized degrades (-19%), reflecting low tool count (2 tools) where coordination overhead is less justified.

    “(f ) Terminal-Bench shows mixed results: Independent achieves marginal gains (+2%) while Centralized degrades (-19%), reflecting the low tool count (2 tools) where coordination overhead is less justified.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 14

    On Terminal-Bench, Independent shows marginal gains (+1.7%, 0.350) while Centralized degrades substantially (-19.2%, 0.278).

    “On Terminal-Bench (SAS mean 0.344, below the threshold), results are mixed: Independent shows marginal gains ( + 1 . 7%, 0.350) while Centralized degrades substantially ( -19 . 2%, 0.278), suggesting that the low tool count (2 tools) limits the benefit of orchestration-heavy architectures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 22

    Anthropic models uniquely benefit from heterogeneous mixing in centralized architecture, where low-capability orchestrator with high-capability subagents (0.42) outperforms homogeneous high-capability (0.32) by 31%.

    “Anthropic models uniquely benefit from heterogeneous mixing in centralized architecture, where low-capability orchestrator with high-capability subagents (0.42) outperforms homogeneous high-capability (0.32) by 31%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 23

    Under-coordination (O < 100% overhead) yields minimal accuracy gain (ΔS approximately +2-4%), with coordination mechanisms not yet engaged.

    “Under-coordination ( 𝑂 &lt; 100% overhead): minimal accuracy gain ( Δ 𝑆 ≈ + 2-4%), coordination mechanisms not yet engaged”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 21

    Beyond the plateau, additional messages yield diminishing returns: Hybrid systems (515% coordination overhead, T = 44.3) show -2.4% versus Centralized (285% overhead, T = 27.7), a difference of 1.1% that is not statistically significant (t(178) = 0.61, p = 0.542).

    “Beyond this point, additional messages yield diminishing returns: Hybrid systems (515% coordination overhead, 𝑇 = 44 . 3) shows -2.4% versus Centralized (285% overhead, 𝑇 = 27 . 7), a difference of 1.1% that is not statistically significant ( 𝑡 ( 178 ) = 0 . 61, 𝑝 = 0 . 542).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    The scaling principle identifies a weak positive interaction between redundancy and agent count (βR×na=0.024, 95% CI [0.002,0.047], p=0.034), suggesting redundancy offers error-correction benefits when more agents participate.

    “The scaling principle identifies a weak positive interaction with agent count ( ˆ 𝛽𝑅 × 𝑛𝑎 = 0 . 024, 95% CI: [ 0 . 002 , 0 . 047 ] , 𝑝 = 0 . 034), suggesting redundancy offers error-correction benefits when more agents participate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 23

    The optimal band (200% < O < 300% overhead) has the highest success-cost ratio (Ec approximately 0.16), dominated by Centralized and Decentralized, with strong error absorption.

    “Optimal band (200% &lt; 𝑂 &lt; 300% overhead): highest success-cost ratio ( 𝐸𝑐 ≈ 0 . 16), dominated by Centralized and Decentralized, with strong error absorption”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    The scaling principle reveals overhead interacts with tool count (βO%×T = -0.033, p=0.211), a directional pattern that loses significance in the 6-benchmark model due to increased heterogeneity.

    “The scaling principle reveals this overhead interacts with tool count ( ˆ 𝛽𝑂 % × 𝑇 = -0 . 033, 𝑝 = 0 . 211), a directional pattern that loses significance in the 6-benchmark model due to the increased heterogeneity of task domains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 15

    In degrading domains, architecture selection becomes a least-worst optimization: PlanCraft shows Hybrid as relatively best (-39.1%) compared to MAS-Centralized (-50.3%) and MAS-Independent (-70.0%).

    “In degrading domains, architecture selection becomes a least-worst optimization: PlanCraft shows Hybrid as relatively best (-39.1%) compared to MAS-Centralized (-50.3%) and MAS-Independent (-70.0%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    Redundancy provides marginal benefit at scale (βR×na=0.024, p=0.034): work redundancy ranges from 0.41 (centralized) to 0.50 (decentralized) for multi-agent systems.

    “Redundancy Provides Marginal Benefit at Scale ( ˆ 𝛽𝑅 × 𝑛𝑎 = 0 . 024 , 𝑝 = 0 . 034 ). Work redundancy, defined as the fraction of subtasks performed by multiple agents, ranges from 0.41 (centralized) to 0.50 (decentralized) for multi-agent systems (Table 5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 17

    Using Intelligence Index, model achieves R2 train=0.463, R2 CV=0.373 (±0.170 SD); replacing with ACI improves fit to R2 train=0.481, R2 CV=0.413 (±0.130 SD), AIC=-244.8, with no reversals in statistical significance across predictors.

    “Using the Intelligence Index as the capability metric, the model achieves 𝑅 2 train = 0 . 463, 𝑅 2 CV = 0 . 373 ( ± 0 . 170 SD). Replacing the Intelligence Index with the task-grounded Agentic Capability Index (ACI), defined as each model's mean single-agent performance across all six benchmarks (correlation with Intelligence Index: 𝑟 = 0 . 45), improves model fit to 𝑅 2 train = 0 . 481, 𝑅 2 CV =”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 16

    The regression model achieves cross-validated R2 = 0.413 with ACI and R2 = 0.373 with Intelligence Index.

    “Regression Model Achieves Cross-Validated 𝑅 2 = 0 . 413 (ACI) / 𝑅 2 = 0 . 373 (Intelligence Index).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4.3. Scaling principles · p. 19

    For a 4-agent system with R=0.50, the redundancy interaction is equivalent to an ≈5% performance boost (in standardized units).

    “equivalent to an ≈ 5% performance boost (in standardized units).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    After controlling for other coordination metrics, neither the main effect of error amplification (β=0.014, p=0.658) nor its interaction with tool count (A×T: β=0.022, p=0.332) reaches statistical significance.

    “After controlling for other coordination metrics, neither the main effect of error amplification ( ˆ 𝛽 = 0 . 014, 𝑝 = 0 . 658) nor its interaction with tool count ( 𝐴 trace 𝑒 × 𝑇 : ˆ 𝛽 = 0 . 022, 𝑝 = 0 . 332) reaches statistical significance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 21

    These architectures achieve 22.7% average error reduction (95% CI: [20.1%, 25.3%]), peaking at 31.4% for Finance Agent where structured numerical outputs facilitate verification.

    “These architectures achieve 22.7% average error reduction (95% CI: [ 20 . 1% , 25 . 3% ] ), peaking at 31.4% for Finance Agent where structured numerical outputs facilitate verification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    Workbench confirms this pattern: decentralized (mean 0.664) outperforms centralized (0.621) despite higher overhead, due to its superior parallel efficiency.

    “Empirically, workbench confirms this pattern: decentralized (mean 0.664) outperforms centralized (0.621) despite higher overhead, due to its superior parallel efficiency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 19

    The model with interaction terms achieves best cross-validated fit (R2 CV = 0.373) and AIC (-236.3), demonstrating empirical coordination metrics capture meaningful variance beyond base predictors alone.

    “Table 3 | Scaling principle model comparison. Progressive inclusion of empirical coordination metrics substantially improves predictive power. All models use 5-fold cross-validation with experiment-level holdout ( 𝑁 = 260, six benchmarks). The full model with interaction terms achieves the best crossvalidated fit ( 𝑅 2 CV = 0 . 373) and AIC ( -236 . 3), demonstrating that empirical coordination”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2. Main Results · p. 15

    On PlanCraft, Anthropic shows maximum -54.5% (MAS-Hybrid 0.31 vs. SAS 0.68), Google shows -25.3% (best), and OpenAI shows -32.3%.

    “On PlanCraft , where all variants degrade, vendor preferences flatten: Anthropic shows maximum -54.5% (MAS-Hybrid 0.31 vs. SAS 0.68), Google shows -25.3% (best), and OpenAI shows -32.3%, indicating that communication mechanisms cannot overcome fundamental sequential reasoning constraints.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 19

    Decentralized architecture exhibits highest redundancy (R=0.50±0.06) and achieves top performance on tool-heavy tasks (workbench success 0.664), consistent with redundancy's protective role.

    “Decentralized architecture, which exhibits highest redundancy ( 𝑅 = 0 . 50 ± 0 . 06), achieves top performance on tool-heavy tasks (workbench success 0.664), consistent with redundancy's protective role.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 17

    The efficiency-tools interaction is a consistent directional pattern: βEc×T = -0.096 (95% CI [-0.154, -0.037], p=0.002), the largest effect size among interaction terms.

    “The Efficiency-Tools Interaction Emerges as a Consistent Directional Pattern ( ˆ 𝛽 = -0 . 096 , 𝑝 = 0 . 002 ). Amongthe significant interactions, the efficiency-tools trade-off exhibits the largest effect size among interaction terms: ˆ 𝛽𝐸𝑐 × 𝑇 = -0 . 096 (95% CI: [-0 . 154 , -0 . 037 ] , 𝑝 = 0 . 002).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    Multi-agent architectures incur substantial overhead: independent 58%, centralized 285%, decentralized 263%, and hybrid 515%, representing 1.6-6.2× token budgets relative to single-agent at matched performance.

    “Multi-agent architectures incur substantial overhead: independent (58%), centralized (285%), decentralized (263%), and hybrid (515%), representing 1.6-6.2 × token budgets relative to single-agent at matched performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 21

    Performance plateaus near c* = 0.39 messages/turn (achieved by Decentralized and Centralized architectures at 0.41 and 0.39 respectively), corresponding to success rates of 47.7% and 46.3%.

    “Performance plateaus near 𝑐 ∗ = 0 . 39 messages/turn (achieved by Decentralized and Centralized architectures at 0.41 and 0.39 respectively), corresponding to success rates of 47.7% and 46.3%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 19

    Cross-validation on held-out configurations confirms the rule achieves 87% correct architecture selection, substantially exceeding random choice (20%) or capability-only models (54%).

    “Cross-validation on held-out configurations confirms this rule achieves 87% correct architecture selection, substantially exceeding random choice (20%) or capability-only models (54%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 23

    Over-coordination (O > 400% overhead) has Hybrid runs with reduced efficiency (Ec approximately 0.11), protocol complexity introducing coordination-failure modes.

    “Over-coordination ( 𝑂 &gt; 400% overhead): Hybrid runs with reduced efficiency ( 𝐸𝑐 ≈ 0 . 11), protocol complexity introducing coordination-failure modes”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 17

    Single-agent systems achieve Ec=0.466, while multi-agent architectures range from Ec=0.074 (hybrid) to Ec=0.234 (independent), a 2-6× efficiency penalty.

    “Empirically, singleagent systems achieve 𝐸𝑐 = 0 . 466 (Table 5), while multi-agent architectures range from 𝐸𝑐 = 0 . 074 (hybrid) to 𝐸𝑐 = 0 . 234 (independent), a 2-6 × efficiency penalty.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 19

    Decentralized architecture underperforms on planning tasks (0.282), where redundancy becomes wasteful duplication.

    “Yet this same architecture underperforms on planning tasks (0.282), where redundancy becomes wasteful duplication.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3. Scaling principles · p. 18

    Trace-level error amplification factors: single-agent 1.0, centralized 4.4, decentralized 7.8, hybrid 5.1, independent multiagent 17.2.

    “Table 5 reveals dramatic variance in trace-level error amplification factors: single-agent ( 𝐴 trace 𝑒 = 1 . 0), centralized ( 𝐴 trace 𝑒 = 4 . 4), decentralized ( 𝐴 trace 𝑒 = 7 . 8), hybrid ( 𝐴 trace 𝑒 = 5 . 1), and independent multiagent ( 𝐴 trace 𝑒 = 17 . 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 21

    Within-architecture variance in turns remains substantial; at n = 3, Independent averages 11.4 turns versus Decentralized 26.1 turns.

    “within-architecture variance remains substantial (e.g., at n = 3: Independent averages 11.4 turns vs. Decentralized 26.1 turns)”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § B. Out-of-Sample Validation · p. 37

    Best MAS performance converges (0.48-0.50) across the three held-out models with Intelligence Index = 75.

    “However, best MAS performance converges (0 . 48-0 . 50)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § B. Out-of-Sample Validation · p. 36

    Independent MAS degradation validates only for GPT-5.2 but not for Gemini models.

    “Independent MAS degradation validates only for GPT-5.2 but not for Gemini models”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § 5. Limitations · p. 26

    Tool-heavy environments represent a primary failure mode for multiagent coordination, with significant negative interactions between tool count and system efficiency.

    “Our analysis reveals that tool-heavy environments represent a primary failure mode for multiagent coordination, with significant negative interactions between tool count and system efficiency.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § B. Out-of-Sample Validation · p. 37

    Empirically, Centralized and Decentralized architectures achieve superior performance to Hybrid for the three held-out models.

    “yet empirically Centralized and Decentralized architectures achieve superior performance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    Token efficiency (success per 1,000 tokens) is 67.7 for SAS, 21.5 for Centralized (3.1 times worse), 23.9 for Decentralized (2.8 times worse), and 13.6 for Hybrid (5.0 times worse).

    “SAS achieves 67.7 successes/1K tokens; Centralized drops to 21.5 (3.1 × worse); Decentralized to 23.9 (2.8 × worse); Hybrid to 13.6 (5.0 × worse).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § B. Out-of-Sample Validation · p. 37

    Despite identical Intelligence Index = 75, single-agent performance varies substantially: GPT-5.2 achieves P_SA = 0.45, while Gemini-3.0 Pro and Gemini-3.0 Flash achieve 0.36 and 0.34, respectively.

    “Despite Index = 75, single-agent performance varies substantially: GPT-5.2 achieves 𝑃 SA = 0 . 45, while Gemini-3.0 Pro and Gemini-3.0 Flash achieve 0.36 and 0.34, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § B. Out-of-Sample Validation · p. 36

    Across three held-out models, 11 of 15 finding-model pairs validate (73%), with two additional partial validations.

    “Across three models, 11 of 15 finding-model pairs validate (73%), with two additional partial validations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § 5. Limitations · p. 26

    A preliminary investigation of 13 heterogeneous configurations on BrowseComp-Plus finds no evidence that model mixing bypasses the capability-saturation threshold: centralized heterogeneous configurations underperform their strong-model homogeneous counterparts by a mean of 12.6 percentage points, while decentralized configurations show marginal gains (+2.0 pp) largely attributable to the stronger

    “A preliminary investigation of 13 heterogeneous configurations on BrowseComp-Plus (mixing models within and across families) finds no evidence that model mixing bypasses the capabilitysaturation threshold: centralized heterogeneous configurations underperform their strong-model homogeneous counterparts by a mean of 12.6 percentage points, while decentralized configurations show marginal gains (+2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § F. SWE-bench Verified and Terminal-Bench Results · p. 42

    The Agentic Capability Index (ACI) achieves the best cross-validated fit with zero finding reversals.

    “The Agentic Capability Index (ACI) achieves the best cross-validated fit with zero finding reversals.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6. Conclusion · p. 27

    Two additional directional patterns emerge consistently across all six benchmarks: a tool-coordination trade-off where tool-heavy tasks suffer from coordination overhead, and architecture-dependent trace-level error amplification ranging from 4.4 times (centralized) to 17.2 times (independent).

    “Two additional directional patterns emerge consistently across all six benchmarks: a tool-coordination trade-off where tool-heavy tasks suffer from coordination overhead, and architecture-dependent trace-level error amplification ranging from 4.4 × (centralized) to 17.2 × (independent).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    Anthropic Hybrid reaches approximately $0.024 per 1% gain (3 times worse than OpenAI).

    “Anthropic Hybrid reaches ≈ $0 . 024 per 1% gain (3 × worse”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6. Conclusion · p. 27

    The paper derives a predictive model (R2 = 0.373 across all six benchmarks; R2 = 0.413 with a task-grounded capability metric) that achieves 87% accuracy in selecting optimal architectures for held-out configurations.

    “We derive a predictive model ( 𝑅 2 = 0 . 373 across all six benchmarks; 𝑅 2 = 0 . 413 with a task-grounded capability metric) that achieves 87% accuracy in selecting optimal architectures for held-out configurations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6. Conclusion · p. 27

    Capability saturation is the most robust scaling effect: coordination yields diminishing returns beyond approximately 45% single-agent baselines, confirmed under both cluster-robust inference (p = 0.004) and Holm-Bonferroni multiple-comparison correction (pHolm = 0.018).

    “We identify capability saturation as the most robust scaling effect: coordination yields diminishing returns beyond ∼ 45% single-agent baselines, a finding confirmed under both cluster-robust inference ( 𝑝 = 0 . 004) and Holm-Bonferroni multiple-comparison correction ( 𝑝 Holm = 0 . 018).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6. Conclusion · p. 27

    Performance gains vary substantially by task structure, from +80.8% on Finance Agent to -70.0% on PlanCraft, demonstrating that coordination benefits depend on task decomposability rather than team size.

    “Performance gains vary substantially by task structure, from +80.8% on Finance Agent to -70.0% on PlanCraft, demonstrating that coordination benefits depend on task decomposability rather than team size.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    The domain-dependent information-gain pattern maps to observed MAS benefits: Finance Agent up to +80.8% for Centralized; BrowseComp-Plus up to +9.2% for Decentralized.

    “This domain-dependent information-gain pattern directly maps to observed MAS benefits: Finance Agent (up to +80.8% for Centralized) where information exchange is high-value; BrowseComp-Plus (up to +9.2% for Decentralized) where world ambiguity limits verification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § B. Out-of-Sample Validation · p. 37

    The equation systematically over-predicts Hybrid performance (mean error +37.0%) while achieving reasonable calibration for Centralized (mean error +3.0%) and Decentralized (signed mean error +8.8%).

    “The equation systematically over-predicts Hybrid performance (mean error + 37 . 0%) while achieving reasonable calibration for Centralized (mean error + 3 . 0%) and Decentralized (signed mean error + 8 . 8%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Findingown data
    § 4.5. Robustness and Sensitivity Analysis · p. 25

    ACI improves cross-validated fit (R2CV: 0.373 to 0.413) with zero finding reversals.

    “ACI improves cross-validated fit ( 𝑅 2 CV : 0 . 373 → 0 . 413) with zero finding reversals”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § F. SWE-bench Verified and Terminal-Bench Results · p. 44

    Terminal-Bench (lower baselines) shows mixed results with low tool count limiting coordination benefits.

    “Terminal-Bench (lower baselines) shows mixed results with low tool count limiting coordination benefits.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    Extrapolation to larger teams via the fitted power law predicts approximately 69 turns at n = 6 and approximately 157 turns at n = 10 (95% CI from exponent uncertainty: [64, 74] and [143, 172] respectively), corresponding to 9.5-21.8 times increases over the SAS baseline of 7.2 turns.

    “Extrapolation to larger teams via the fitted power law predicts ≈ 69 turns at 𝑛 = 6 and ≈ 157 turns at 𝑛 = 10 (95% CI from exponent uncertainty: [64, 74] and [143, 172] respectively), corresponding to 9 . 5 × -21 . 8 × increases over the SAS baseline of 7.2 turns.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The coordination overhead measured by turns or messages is a valid measure of computational cost.

  • May combine two claims.

    Findingown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    OpenAI models show strongest Hybrid gains on structured tasks (Finance: 52% success Hybrid vs. 39% SAS; Workbench: 56% Hybrid vs. 42% SAS).

    “OpenAI models show strongest Hybrid gains on structured tasks (Finance: 52% success Hybrid vs. 39% SAS; Workbench: 56% Hybrid vs. 42% SAS).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 2

    Lossy communication increases synchronization overhead and cognitive load.

    “This lossy communication increases synchronization overhead and cognitive load [29]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [29] [29] Thomas W Malone and Kevin Crowston. The interdisciplinary study of coordination. ACM Computing Surveys ( CSUR) , 26(1):87-119, 1994. - source

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 15

    The critical factor is task decomposability: Finance Agent (D=0.41) achieves +80.8% gains through parallelizable subtask structure, whereas PlanCraft (D=0.42) degrades by -70% due to strict sequential dependencies despite similar complexity scores.

    “While low-complexity domains (Workbench, D = 0.00) show modest gains and high-complexity domains (BrowseComp-Plus, D = 0.84) show limited benefits, the critical factor is task decomposability: Finance Agent (D = 0.41) achieves +80.8% gains through parallelizable subtask structure, whereas PlanCraft (D = 0.42) degrades by -70% due to strict sequential dependencies despite similar complexity scores.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 1. Introduction · p. 4

    Architecture selection is governed by measurable task features such as decomposability rather than simple agent scaling, achieving 87% accuracy in predicting optimal architectures on held-out tasks.

    “Using these mechanisms, we demonstrate that architecture selection is governed by measurable task features (e.g., decomposability) rather than simple agent scaling, achieving 87% accuracy in predicting optimal architectures on held-out tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.2. Agentic Tasks and Benchmarks · p. 9

    Scores are normalized to the best single-agent baseline, measuring coordination gain or loss.

    “Comparative Normalization: scores are normalized to the best single-agent baseline, measuring coordination gain or loss.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.2. Agentic Tasks and Benchmarks · p. 9

    Within each model family, evaluation emphasizes adaptive reasoning over memorized facts.

    “Controlled for Parametric Knowledge: within each model family, evaluation emphasizes adaptive reasoning over memorized facts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction · p. 2

    Recent work has identified cases where single strong models match or exceed multi-agent systems.

    “Recent work has identified cases where single strong models match or exceed multi-agent systems [16]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] [16] Mingyan Gao, Yanzi Li, Banruo Liu, Yifan Yu, Phillip Wang, Ching-Yu Lin, and Fan Lai. Singleagent or multi-agent systems? why not both? arXiv preprint arXiv:2505.18286 , 2025. - source

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 14

    Coordination overhead becomes counterproductive when coordination complexity exceeds task complexity (PlanCraft), but provides substantial gains when tasks naturally decompose into parallel information streams (Finance Agent).

    “These trajectory patterns reveal the mechanistic basis for domain-dependence: coordination overhead becomes counterproductive when coordination complexity exceeds task complexity (PlanCraft), but provides substantial gains when tasks naturally decompose into parallel information streams (Finance Agent).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 14

    SWE-bench Verified degradation is consistent with a capability-saturation threshold: most models achieve single-agent baselines above 45%, leaving limited room for coordination gains.

    “This is consistent with the capability-saturation threshold: most models achieve single-agent baselines above 45%, leaving limited room for coordination gains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 14

    MAS do not provide universal benefits but rather domain-specific trade-offs, with performance range from -70.0% (PlanCraft Independent) to +80.8% (Finance Centralized).

    “The performance range across MAS variants spans from -70 . 0% (PlanCraft Independent) to + 80 . 8% (Finance Centralized), indicating that MAS do not provide universal benefits but rather domain-specific trade-offs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.2. Agentic Tasks and Benchmarks · p. 9

    The benchmarks are chosen such that base LLMs perform poorly in single-shot mode and non-trivial performance requires multi-step environment interaction.

    “Hence, our benchmarks are chosen such that (i) base LLMs perform poorly in single-shot mode, and (ii) non-trivial performance requires multi-step environment interaction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § 1. Introduction · p. 3

    The experimental design isolates architectural effects by controlling for implementation confounds, maintaining identical task prompts, tools, and computational budgets across all configurations, while systematically varying only coordination structure and model capability.

    “Our experimental design isolates architectural effects by controlling for implementation confounds which maintains identical task prompts, tools, and computational budgets across all configurations, while systematically varying only coordination structure and model capability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 4.1. Setup · p. 12

    MAS and SAS systems were matched for maximum total iterations/computational budget, with MAS receiving equal budget through parallel processing and SAS proportionally more reasoning rounds.

    “To ensure computational fairness, we matched maximum total iterations between MAS and SAS systems: MAS configurations received equal computational budget through parallel agent processing (smaller per-agent iterations for 𝑛 -agent teams), while SAS received proportionally more reasoning rounds to compensate for lack of parallel deliberation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 14

    Domain complexity moderates MAS advantage: structured, decomposable domains show large gains while high-complexity sequential domains show consistent degradation.

    “Domain Complexity Moderates Coordination Efficacy. Empirical patterns across benchmarks reveal that domain complexity (refer to Appendix C for details) moderates MAS advantage: structured, decomposable domains show large gains while high-complexity sequential domains show consistent degradation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § 3.2. Agentic Tasks and Benchmarks · p. 9

    Cross-family comparisons account for inherent knowledge base differences through baseline normalization.

    “Cross-family comparisons (Section 4) account for inherent knowledge base differences through baseline normalization.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3. Scaling principles · p. 19

    The redundancy relationship may be context-dependent, potentially stronger in error-prone domains or weaker when communication is expensive.

    “The significance ( 𝑝 = 0 . 034, near the 𝛼 = 0 . 05 threshold) suggests this relationship may be context-dependent, potentially stronger in error-prone domains or weaker when communication is expensive.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3. Scaling principles · p. 19

    The scaling principle constitutes both a scientific contribution and an engineering tool for architecture selection within known task regimes.

    “The scaling principle thus constitutes both a scientific contribution, a cross-domain descriptive framework for relating coordination metrics to agent performance, and an engineering tool for architecture selection within known task regimes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 4.3. Scaling principles · p. 17

    The equation contains no dataset-specific parameters or dataset-dependent tuning, enabling prediction on unseen task domains.

    “This equation contains no dataset-specific parameters or dataset-dependent tuning , enabling prediction on unseen task domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 21

    Under fixed computational budgets, per-agent reasoning capacity becomes prohibitively thin beyond 3-4 agents, creating a hard resource ceiling where communication cost dominates reasoning capability.

    “under fixed computational budgets, per-agent reasoning capacity becomes prohibitively thin beyond 3-4 agents, creating a hard resource ceiling where communication cost dominates reasoning capability”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3. Scaling principles · p. 18

    For hybrid architecture (O%=515) on workbench (T=16), overhead costs compound with tool complexity, explaining hybrid's collapse on tool-heavy benchmarks (success rate 0.452 overall, 0.21 on workbench).

    “The direction is preserved: for hybrid architecture ( 𝑂 % = 515) on workbench ( 𝑇 = 16), overhead costs compound with tool complexity, explaining hybrid's collapse on tool-heavy benchmarks (success rate 0.452 overall, 0.21 on workbench).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3. Scaling principles · p. 18

    Coordination benefits scale consistently with model capability, without evidence of emergent super-linear gains at higher intelligence levels.

    “This finding suggests that coordination benefits scale consistently with model capability, without evidence of emergent super-linear gains at higher intelligence levels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.2. Main Results · p. 15

    No vendor achieves universal multi-agent dominance; each exhibits relative advantages in structured domains (Finance) that evaporate in sequential constraint-satisfaction domains (PlanCraft), indicating multi-agent benefits are contingent on problem structure rather than generalizable across task types.

    “No vendor achieves universal multi-agent dominance; instead, each exhibits relative advantages in structured domains (Finance) that evaporate in sequential constraint-satisfaction domains ( PlanCraft ), indicating that multi-agent benefits are genuinely contingent on problem structure rather than generalizable across task types.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.3. Scaling principles · p. 18

    The finding contradicts the naïve hypothesis that 'more agents always help with complexity': tool-rich environments amplify the coordination tax, making simpler architectures more effective.

    “This finding contradicts the naïve hypothesis that 'more agents always help with complexity': tool-rich environments amplify the coordination tax, making simpler architectures more effective.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 4.3. Scaling principles · p. 17

    Model complexity was validated through five-fold cross-validation with experiment-level holdout.

    “We validate model complexity through five-fold crossvalidation with experiment-level holdout (splitting at the configuration level).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.3. Scaling principles · p. 18

    Independent architecture's universal underperformance (mean success 0.370 vs. 0.466 SAS) stems from absence of inter-agent communication: each agent operates in isolation, duplicating errors without correction opportunities, but this effect is subsumed by the efficiency metric (Ec=0.234 for Independent vs. Ec=0.466 for SAS).

    “Independent architecture's universal underperformance (mean success 0.370 vs. 0.466 SAS) stems from absence of inter-agent communication: each agent operates in isolation, duplicating errors without correction opportunities, but this effect is subsumed by the efficiency metric ( 𝐸𝑐 = 0 . 234 for Independent vs. 𝐸𝑐 = 0 . 466 for SAS).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 20

    All MAS and SAS configurations were matched for total reasoning-token budget (mean 4,800 tokens per trial) and tool-call access to isolate coordination effects.

    “All MAS and SAS configurations were matched for total reasoning-token budget (mean 4,800 tokens per trial) and tool-call access to isolate coordination effects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning
    § 5. Limitations · p. 26

    The six agentic benchmarks may not capture the full spectrum of agentic task characteristics.

    “may not capture the full spectrum of agentic task characteristics”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    All experiments use identical tool interfaces and observation structures across architectures to eliminate external feedback confounds.

    “All experiments use identical tool interfaces and observation structures across architectures to eliminate external feedback confounds.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § A. Model Intelligence Index · p. 35

    The reconstructed Intelligence Index values are methodologically consistent but not officially certified estimates.

    “These reconstructed values should be interpreted as methodologically consistent but not officially certified estimates.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    Context window management applies consistent truncation policies.

    “Context window management applies consistent truncation policies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § A. Model Intelligence Index · p. 35

    Several evaluations, particularly long-context and agentic workflow tasks, contain nondeterministic components that may vary slightly across implementations.

    “First, several evaluations, particularly long-context and agentic workflow tasks, contain nondeterministic components that may vary slightly across implementations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § A. Model Intelligence Index · p. 35

    The Intelligence Index evaluates all models under consistent, zero-shot, instruction-prompted conditions and employs pass@1 scoring and robust equality-checker mechanisms.

    “The Intelligence Index meets this requirement by: (i) evaluating all models under consistent, zero-shot, instruction-prompted conditions; (ii) employing pass@1 scoring and robust equality-checker mechanisms;”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 5. Limitations · p. 27

    The regression analysis clusters observations at the dataset level (G = 6).

    “Our regression analysis clusters observations at the dataset level ( 𝐺 = 6).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    Bootstrap 95% confidence intervals are reported in Table 16.

    “bootstrap 95% confidence intervals are reported in Table 16.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § A. Model Intelligence Index · p. 35

    The Artificial Analysis Intelligence Index provides a publicly available synthesis of model capabilities, combining performance across reasoning, knowledge, mathematics, coding, instruction following, long-context reasoning, and agentic workflow tasks.

    “This index provides a publicly available synthesis of model capabilities, combining performance across reasoning, knowledge, mathematics, coding, instruction following, long-context reasoning, and agentic workflow tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    Instance selection ensures representative coverage of task types and difficulty levels within each benchmark.

    “Instance selection ensures representative coverage of task types and difficulty levels within each benchmark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § B. Out-of-Sample Validation · p. 37

    The discrepancy between predicted Hybrid optimality and empirical Centralized/Decentralized superiority reflects linear extrapolation of the intelligence effect (β_I = 0.126) beyond its training range and the model's failure to capture Hybrid's disproportionate overhead penalty at high capability levels.

    “This discrepancy reflects two factors: (1) linear extrapolation of the intelligence effect ( ˆ 𝛽𝐼 = 0 . 126) beyond its training range, and (2) the model's failure to capture Hybrid's disproportionate overhead penalty at high capability levels.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 5. Limitations · p. 26

    Prompts were controlled to be identical across conditions for experimental validity.

    “While we controlled prompts to be identical across conditions for experimental validity”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    Evaluation uses frozen model weights without fine-tuning to measure architectural effects independently of model optimization.

    “Evaluation uses frozen model weights without fine-tuning to measure architectural effects independently of model optimization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning
    § A. Model Intelligence Index · p. 35

    The Intelligence Index reports a composite measure reflecting general-purpose reasoning and problem-solving ability and demonstrates high statistical reliability (reported confidence interval below ±1%).

    “(iii) reporting a composite measure reflecting general-purpose reasoning and problem-solving ability; and (iv) demonstrating high statistical reliability (reported confidence interval below ± 1%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § E.4. Evaluation Methodology · p. 41

    API rate limiting and retry mechanisms ensure fair resource allocation.

    “API rate limiting and retry mechanisms ensure fair resource allocation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § B. Out-of-Sample Validation · p. 37

    Intelligence Index, while predictive within model families, may not be directly comparable across vendors.

    “This implies that Intelligence Index, while predictive within model families, may not be directly comparable across vendors, a limitation for cross-family extrapolation that future scaling laws should address.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

  • May combine two claims.

    Study design and conductnone given
    § 5. Limitations · p. 27

    With a small number of clusters, cluster-robust standard errors are known to be conservative, and several predictors that are significant under naive OLS lose significance under cluster-robust inference.

    “With a small number of clusters, cluster-robust standard errors are known to be conservative, and several predictors that are significant under naive OLS lose significance under cluster-robust inference”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § A. Model Intelligence Index · p. 35

    Artificial Analysis maintains private test variants and additional filtering procedures that cannot be fully reproduced.

    “Third, Artificial Analysis maintains private test variants and additional filtering procedures that cannot be fully reproduced.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 5. Limitations · p. 26

    SWE-bench Verified and Terminal-Bench use 20-instance subsets due to the computational cost of Docker-based evaluation environments.

    “SWE-bench Verified and Terminal-Bench use 20-instance subsets (smaller than the 50-100 instances used for BrowseComp-Plus, Finance Agent, PlanCraft, and Workbench) due to the computational cost of Docker-based evaluation environments.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 5. Limitations · p. 26

    Prompts were not optimized specifically for each model or model family.

    “we did not optimize prompts specifically for each model or model family”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § 5. Limitations · p. 26

    Bootstrap 95% confidence intervals yield typical widths of plus or minus 20 percentage points per cell; individual pairwise comparisons are underpowered at n = 20, but aggregate trends across 8 models x 2 benchmarks remain robust.

    “Bootstrap 95% confidence intervals (10,000 resamples) yield typical widths of ± 20 percentage points per cell; while individual pairwise comparisons are underpowered at 𝑛 = 20, aggregate trends across 8 models × 2 benchmarks remain robust”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    Agents successfully exchange high-value information and synthesize it into improved solutions.

    “indicating that agents successfully exchange high-value information and synthesize it into improved solutions”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § E.4. Evaluation Methodology · p. 41

    The authors evaluate on dataset subsets balancing computational cost with statistical significance.

    “We evaluate on dataset subsets balancing computational cost with statistical significance: Finance Agent (50 instances), BrowseComp Plus (100 instances), WorkBench (100 instances), Plancraft (100 instances), SWE-bench Verified (20 instances, deterministic shuffle with seed 42 from 500), and Terminal-Bench (20 instances, first𝑁 from 86).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.4. Coordination Efficiency, Error Dynamics, and Information Transfer · p. 24

    Super-linear scaling confirms the hard resource ceiling: beyond 3-4 agents, per-agent reasoning quality degrades sharply under fixed budgets.

    “This super-linear scaling confirms the hard resource ceiling: beyond 3-4 agents, per-agent reasoning quality degrades sharply under fixed budgets.”
    Quote in the paperNot a substantial claim
114 of 114

  • The paper relies on this without support.

    There remains no principled quantitative framework for predicting when adding agents improves performance and when it instead introduces coordination costs that degrade it.

    “Despite rapid adoption, there remains no principled quantitative framework for predicting when adding agents improves performance and when it instead introduces coordination costs that degrade it.”

  • The paper relies on this without support.

    Existing MAS evaluations compare architectures using different prompts, tools, or computational budgets, conflating architectural effects with implementation choices and precluding clean causal attribution.

    “First , existing MAS evaluations compare architectures using different prompts, tools, or computational budgets, conflating architectural effects with implementation choices and precluding clean causal attribution.”

  • The paper relies on this without support.

    Evaluations focus exclusively on final accuracy metrics without examining process dynamics such as coordination overhead, error propagation, and information flow that determine whether collaboration succeeds or fails.

    “Second , evaluations focus exclusively on final accuracy metrics without examining process dynamics such as coordination overhead, error propagation, and information flow that determine whether collaboration succeeds or fails.”

  • The paper relies on this without support.

    Multi-agent system evaluations have been conducted predominantly on non-agentic tasks.

    “multi-agent system evaluations have been conducted predominantly on non-agentic tasks”

  • The paper relies on this without support.

    The evaluation literature provides limited guidance on what factors determine collaborative success, whether semantic diversity predicts team performance, how architectural choices shape coordination costs, or whether agents can detect and correct failures in extended interactions.

    “yet the evaluation literature provides limited guidance on what factors determine collaborative success, whether semantic diversity predicts team performance, how architectural choices shape coordination costs, or whether agents can detect and correct failures in extended interactions.”

  • The paper relies on this without support.

    We lack comparable empirical understanding of how human team principles translate to artificial agents.

    “Yet we lack comparable empirical understanding of how these principles translate to artificial agents”

  • The paper relies on this without support.

    The four coordination topologies (SAS, Independent, Centralized, Decentralized, Hybrid) adequately represent the space of multi-agent architectures relevant to the study's conclusions.

    “Rather than attempting exhaustive coverage of all possible architectures, we selected these four MAS configurations to form a structured ablation over two key coordination dimensions: (i) orchestrator presence (hierarchical control vs. flat structure), and (ii) peer communication (direct sub-agent interaction vs. isolated execution).”

  • The paper relies on this without support.

    The domain complexity score D captures the degree of sequential interdependence and empirical difficulty relevant to MAS effectiveness.

    “We characterize each benchmark by a domain complexity score 𝐷 ∈ [ 0 , 1 ] (Appendix C), capturing the degree of sequential interdependence and empirical difficulty:”

  • The paper relies on this without support.

    The benchmarks and task success/accuracy metrics validly measure agentic performance across domains.

    “Primary outcome is task success/accuracy (domain-dependent: factual correctness for Finance Agent, task completion for Workbench, goal satisfaction for PlanCraft, page synthesis accuracy for BrowseComp-Plus).”

  • The paper relies on this without support.

    Token-overlap metrics (shared-token entropy and contradictory mass) validly indicate message consensus and error correction.

    “The correction mechanism is revealed through token-overlap analysis.”
    “evidence that messages converge toward mutually consistent sub-proofs rather than self-reinforcing errors”

  • The paper relies on this without support.

    Bayesian posterior variance reduction on key variables is a valid surrogate for information gain in these tasks.

    “We compute information gain Δ I by comparing pre-coordination and post-coordination task-uncertainty surrogates (via Bayesian posterior variance reduction on key variables).”

  • The paper relies on this without support.

    The four error categories (logical contradiction, numerical drift, context omission, coordination failure) exhaustively and validly capture agent failure modes.

    “We identified four error categories as follows.”

  • The paper relies on this without support.

    The fitted power law from n <= 9 can be extrapolated to n = 6 and n = 10 to predict turns.

    “Extrapolation to larger teams via the fitted power law predicts ≈ 69 turns at 𝑛 = 6 and ≈ 157 turns at 𝑛 = 10”

  • The paper relies on this without support.

    The coordination overhead measured by turns or messages is a valid measure of computational cost.

    “the communication overhead we measured grows superlinearly with agent count”

  • The paper relies on this without support.

    The Artificial Analysis Intelligence Index is a valid measure of a model's baseline capabilities independent of agentic mechanism.

    “This makes it suitable as a foundational axis for studying how agentic performance scales with underlying model capacity .”

  • The paper relies on this without support.

    The reconstructed Intelligence Index values are sufficiently accurate for use in the scaling analysis.

    “These reconstructed values should be interpreted as methodologically consistent but not officially certified estimates.”

  • The paper relies on this without support.

    Domain complexity can be validly captured by an ordinal score D ∈ [0,1] based on sequential interdependence, coordination overhead sensitivity, and state-space complexity.

    “We characterize domain complexity through an ordinal score 𝐷 ∈ [ 0 , 1 ] that captures the degree of sequential interdependence and empirical difficulty across evaluated benchmarks.”
17 of 17

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Coordination overhead is a first-order constraint.

    “On the other hand, Kim et al. (2025) shows that once computation is normalized, agentic benefits tend to concentrate in weaker model or harder-regime settings and diminish (or even reverse) as base model capability increases, highlighting coordination overhead as a first-order constraint.”

    From Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets (2026) · cites this paper as Kim et al., 2025

  • Once computation is normalized, agentic benefits tend to concentrate in weaker model or harder-regime settings and diminish or even reverse as base model capability increases.

    “On the other hand, Kim et al. (2025) shows that once computation is normalized, agentic benefits tend to concentrate in weaker model or harder-regime settings and diminish (or even reverse) as base model capability increases, highlighting coordination overhead as a first-order constraint.”

    From Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets (2026) · cites this paper as Kim et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Anthropic heterogeneous comparison uses conflicting homogeneous baselines

    Major

    Figure 4 says the Anthropic heterogeneous score of 0.42 exceeds its homogeneous high-capability baseline of 0.32. Table 12 reports the same 0.42 configuration as 0.01 below its strong-model homogeneous baseline, implying a baseline of 0.43.

  • Reported token-budget increase conflicts with matched-token design

    Major

    The results describe multi-agent architectures as receiving up to 6.2 times the single-agent token budget, while Table 5 states that all systems were matched for total reasoning tokens.

  • Results table includes a model excluded from the stated 260 configurations

    Major

    The paper says SWE-bench Verified and Terminal-Bench each use eight models because their model families are unchanged except for the removal of Claude Sonnet 3.7. Table 16 nevertheless includes gemini-3-flash on both benchmarks, giving nine model rows per benchmark and conflicting with the stated configuration count.

  • Hybrid Workbench success is reported as both 0.21 and approximately 0.621

    Major

    The reported 0.21 Workbench success rate conflicts with the earlier statement that Hybrid is only 1.2% below the 0.629 single-agent baseline, which implies approximately 0.621.

  • Decentralized PlanCraft score changes from 0.332 to 0.282

    Minor

    The main results assign 0.332 to Decentralized PlanCraft. A later discussion assigns the same architecture 0.282, the score previously assigned to Centralized.

  • Decentralized Workbench overhead is called higher than Centralized overhead

    Minor

    The paper attributes Decentralized's Workbench advantage to performance despite higher overhead. Table 5 reports the opposite ordering: 263% for Decentralized and 285% for Centralized.

  • Bonferroni-adjusted intelligence p-value does not match 19 tests

    Minor

    With the stated 19 hypotheses, multiplying the reported raw p-value of 0.008 by 19 gives 0.152, not the tabulated Bonferroni p-value of 0.066. The difference cannot be explained by rounding the raw value to three decimals.

    Reported 0.066 · recomputed 0.152

  • Claim of best AIC conflicts with model-comparison table

    Minor

    Table 3 calls the full model's AIC of -236.3 the best, but the coordination-structure model has a lower AIC of -243.2.

8 of 8

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. [9][9] Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Jordan W Suchow, Denghui Zhang, and Khaldoun Khashanah. Finmem: A performance-enhanced LLM trading agent with layered memory and character design. IEEE Transactions on Big Data , 2025. -
  2. [27][27] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning , 2023. -
  3. [31][31] Patrick M Lencioni. The five dysfunctions of a team: A leadership fable . John Wiley &amp; Sons, 2002. -
  4. [35][35] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=WE\_v luYUL-X . -
  5. [41][41] Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. URL https://openreview .net/forum?id=bgzUSZ8aeg . -
  6. [47][47] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/f orum?id=VTF8yNQM66 . -
  7. [49][49] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ . -
  8. [51][51] Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Melroy Maben, Raj Mehta, Wayne Chi, Lawrence Keunho Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2025. URL https://openreview.net/forum?id=LZnKNApvhG . -
  9. [57][57] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling , 2024. -
  10. [59][59] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement , 20(1):37-46, 1960. -

Cited by

Sorted by citations

Authors
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., Malhotra, M., Liang, P. P., Park, H. W., Yang, Y., Xu, X., Du, Y., Patel, S., Althoff, T., McDuff, D., & Liu, X.
Affiliations
  • Google (United States) (US)
  • Massachusetts Institute of Technology (US)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2512.08296
arXiv
2512.08296
OpenAlex
W7114795442
Semantic Scholar
628dcc7046ea462b5e770a0f070c5b0988aaeee0
Cited by
1
References
51
Citations by year
2026: 1
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
multi-agent systems, scaling laws, multi-agent coordination, centralized coordination, distributed coordination, independent agents, error propagation, LLM agents, financial reasoning, sequential planning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review