Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

“This distributed approach mirrors successful patterns in traditional computing—from MapReduce to modern distributed databases—where data partitioning and coordinated computation across nodes achieve scales unattainable by a single machine (Dean and Ghemawat, 2008).”

“The rapid advancement of Large Language Models (LLMs) has demonstrated remarkable capabilities in individual inference and generation tasks (AI, 2023; Touvron et al., 2023; DeepSeek-AI et al., 2024).”

“Existing multi-agent benchmarks either prescribe fixed communication structures (Li et al., 2023; Wu et al., 2024; Hong et al., 2023) or focus on social simulation rather than computational collaboration (Park et al., 2023; Lan et al., 2024).”

“…recent advances have extended context lengths to millions of to-kens (Reid et al., 2024; Liu et al., 2024a), the quadratic computational complexity of attention mechanisms makes centralized processing increasingly resource-intensive and prone to “lost-in-the-middle” phenomena (Liu et al., 2024b).”

“The theoretical foundation for analyzing distributed computation costs dates back to Yao’s seminal work on communication complexity (Yao, 1979), which established the framework for quantifying the minimum bits required for distributed parties to compute a function.”

“Existing multi-agent benchmarks either prescribe fixed communication structures (Li et al., 2023; Wu et al., 2024; Hong et al., 2023) or focus on social simulation rather than computational collaboration (Park et al., 2023; Lan et al., 2024).”

“Foundational works like CAMEL (Li et al., 2023) and MetaGPT (Hong et al., 2023) utilize role-specialized agents (e.g., assigning “Manager” or “Coder” personas) embedded within fixed hierarchical or waterfall workflows.”

“Spanning diverse protocols and computing paradigms (Zhao et al., 2024; Islam et al., 2024a), we pro-pose a multi-dimensional metric suite to comprehensively quantify the trade-off between task success rate, token consumption, and communication density.”

“However, as the scale and complexity of real-world problems continue to grow, a fundamental bottleneck has emerged: the limited context window of a single model restricts its ability to process global information (Li et al., 2024; Chen et al., 2024b; An et al., 2024).”

“Multi-Agent Systems (MAS) offer a compelling architectural paradigm to address this scalability challenge (Zhang et al., 2024a; Wang et al., 2024b).”

“Multi-Agent Systems (MAS) offer a compelling architectural paradigm to address this scalability challenge (Zhang et al., 2024a; Wang et al., 2024b).”

“…fixed roles and communication scripts, our framework can generate unlimited evaluation instances while providing high-level task hints—allowing observation of whether agents can translate structural understanding into effective coordination protocols (Press et al., 2021; Liu et al., 2023).”

“Other efforts, such as debate-based systems (Du et al., 2023) or Mixture-of-Agents (Wang et al., 2024a), often prescribe static topological constraints that limit dynamic information flow.”

“Existing multi-agent benchmarks either prescribe fixed communication structures (Li et al., 2023; Wu et al., 2024; Hong et al., 2023) or focus on social simulation rather than computational collaboration (Park et al., 2023; Lan et al., 2024).”

“Furthermore, most benchmarks to date target specific tasks (Deng et al., 2024; Chen et al., 2024a; Gioacchini et al., 2024) and fail to address a significant gap in our understanding: whether logic-based models can autonomously discover and execute effective coordination strategies for distributed…”

“Although Retrieval-Augmented Generation (RAG) offers a palliative solution (Wang et al., 2024c; Islam et al., 2024b), it often fractures global context, struggling with tasks that require holistic reasoning across disjoint segments.”

“Existing benchmarks like SCROLLS (Shaham et al., 2022), LongBench (Bai et al., 2024), and ∞ Bench (Zhang et al., 2024b) effectively evaluate single-agent retrieval but over-look the paradigm of distributed collaboration .”

“These approaches often introduce inductive bias into the agents’ final outputs (Baltaji et al., 2024).”

“Other efforts, such as debate-based systems (Du et al., 2023) or Mixture-of-Agents (Wang et al., 2024a), often prescribe static topological constraints that limit dynamic information flow.”

“Spanning diverse protocols and computing paradigms (Zhao et al., 2024; Islam et al., 2024a), we pro-pose a multi-dimensional metric suite to comprehensively quantify the trade-off between task success rate, token consumption, and communication density.”

“While effective for domain-specific tasks like software engineering (Islam et al., 2024a), these approaches entangle the agents’ reasoning capabilities with semantic role priors, making it difficult to isolate the contribution of the communication architecture itself.”

“While recent advances have extended context lengths to millions of to-kens (Reid et al., 2024; Liu et al., 2024a), the quadratic computational complexity of attention mechanisms makes centralized processing increasingly resource-intensive and prone to “lost-in-the-middle” phenomena (Liu et al.,…”

“Other efforts, such as debate-based systems (Du et al., 2023) or Mixture-of-Agents (Wang et al., 2024a), often prescribe static topological constraints that limit dynamic information flow.”

“Existing benchmarks like SCROLLS (Shaham et al., 2022), LongBench (Bai et al., 2024), and ∞ Bench (Zhang et al., 2024b) effectively evaluate single-agent retrieval but over-look the paradigm of distributed collaboration .”

“Even with recent progress in extending context lengths to millions of tokens (Reid et al., 2024), the quadratic cost of attention (Ratner et al., 2023; Yen et al., 2024) makes * † Corresponding Author. centralized processing increasingly impractical for truly large-scale tasks.”

“Although Retrieval-Augmented Generation (RAG) offers a palliative solution (Wang et al., 2024c; Islam et al., 2024b), it often fractures global context, struggling with tasks that require holistic reasoning across disjoint segments.”

“For instance, if an agent is assigned the role of a "doctor", it may exhibit poor performance in artistic domains, which contra-dicts the goal of developing general-purpose agents (An et al., 2024; Qian et al., 2024, 2023; Liu et al., 2023).”

“Partial Correctness Score ( P ).”

“Existing benchmarks like SCROLLS (Shaham et al., 2022), LongBench (Bai et al., 2024), and ∞ Bench (Zhang et al., 2024b) effectively evaluate single-agent retrieval but over-look the paradigm of distributed collaboration .”

“Furthermore, most benchmarks to date target specific tasks (Deng et al., 2024; Chen et al., 2024a; Gioacchini et al., 2024) and fail to address a significant gap in our understanding: whether logic-based models can autonomously discover and execute effective coordination strategies for distributed…”

“…to date target specific tasks (Deng et al., 2024; Chen et al., 2024a; Gioacchini et al., 2024) and fail to address a significant gap in our understanding: whether logic-based models can autonomously discover and execute effective coordination strategies for distributed computing problems.”

“Silo-Bench similarly shows that agents may exchange enough information but fail to integrate it into a correct joint answer (Zhang et al., 2026).”

“In the LLM era, COMMA [10] evaluated communicative collaboration among multimodal agents under information asymmetry, MultiAgentBench [14] measured cooperative and competitive behaviors in scenarios such as Minecraft, and SILO-BENCH [15] tested multi-agent coordination under distributed information.”

“We also evaluate on Silo-Bench-style tasks [24], where agents operate under information silos and must coordinate to recover a global answer.”

“AgentBench, GAIA, AgentBoard, WebArena, SWE-bench, Tool-Bench, and Silo-Bench [16, 22–24, 29, 40, 41] extend agent evaluation to interaction, tool use, web/software tasks, analytical traces, and distributed coordination.”

“LLMs similarly withhold unshared information (Li et al., 2026; Zhang et al., 2026; Yadav et al., 2026; Cemri et al., 2025).”

“This deterioration is commonly attributed to uneven task distribution and communication barriers among agents 204,205 .”

“…[23] Task interface and evaluator Task-specific responsibilities and coordination Cross-domain transfer of the design procedure SILO-BENCH [24] Task generator and per-agent shard size Agent scale and communication protocol Information distribution and scalability under silos TeamBench…”

“For SILO-BENCH, if an agent produces no final answer, we ask it to answer once more with thinking disabled.”

“MultiAgent-Bench [1], SILO-BENCH [30], and COMMA [31] characterize how topology, roles, and communication policy shape collaboration; NetSafe [32] and G-Safeguard [33] study topology-dependent exposure and intervention.”

“Benchmarks that do stress coordination exist, built around long-horizon tool use, siloed state, or distributed evidence (Gao et al., 2026; Zhang et al., 2026c; Chen et al., 2026a).”

“More generally, agentic protocols can themselves introduce execution overhead or alter model behavior beyond the underlying model capability (Zhang et al., 2026a), motivating our explicit comparison of execution protocols rather than treating the surrounding agent framework as neutral.”

“…frontier agents across more than sixteen hundred runs communicate actively yet fail to turn that communication into a correct joint result, and the failure deepens as the required coordination grows, reaching zero success on the hardest tasks at fifty or more agents (Zhang et al., 2026).”

“Decentralized agent networks (Yang et al., 2025; Li et al., 2025; Ruan et al., 2025; Zhang et al., 2026b; Tian et al., 2026) and consensus schemes robust to Byzantine agents (Jo and Park, 2025; Lee et al., 2026) use frozen models.”

“For example, GEMMAS introduces Unnecessary Path Ratio (UPR) to quantify redundant reasoning paths [49], while SILO-BENCH introduces Relative Coordination Cost (RCC) to measure performance lost to coordination overhead relative to a single-agent baseline as team size increases [50].”

“Failure tax-onomies characterize when coordination breaks down—agents misalign on roles, knowledge, or intentions—from annotated execution traces or controlled fault injection [5]–[7].”

“Making information available through additional routes helps only if recipients can use it in their decisions (Jiang et al., 2026; Zhang et al., 2026).”

“HiddenBench and Silo-Bench test whether agents can communicate and combine information from separate contexts (Li, Naito, and Shirado 2025; Zhang et al. 2026).”

“Future research should investigate scalable frameworks for continuous data collection from real users in production environments (Kong et al., 2025; Zhang et al., 2026a), while addressing the inherent challenges of data quality control, annotation efficiency, and temporal distribution shifts.”