Deep reinforcement learning for multi-agent coordination
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.
Abstract
We address the challenge of coordinating multiple robots in narrow and confined environments, where congestion and interference often hinder collective task performance. Drawing inspiration from insect colonies, which achieve robust coordination through stigmergy—modifying and interpreting environmental traces—we propose a stigmergic multi-agent deep reinforcement learning (S-MADRL) framework that leverages virtual pheromones to model local and social interactions, enabling decentralized emergent coordination without explicit communication. To overcome the convergence and scalability limitations of existing algorithms such as MADQN, MADDPG, and MAPPO, we leverage curriculum learning, which decomposes complex tasks into progressively harder sub-problems. Simulation results show that our framework achieves the most effective coordination of up to eight agents, where robots self-organize into asymmetric workload distributions that reduce congestion and modulate group performance. This emergent behavior, analogous to strategies observed in nature, demonstrates a scalable solution for decentralized multi-agent coordination under crowded environments with communication constraints.
Overview
CoChat Summary
TL;DR
The paper proposes S-MADRL, a decentralized multi-agent deep RL framework that uses virtual pheromone stigmergy and curriculum learning to coordinate homogeneous robots in a narrow pellet-retrieval task. It reports stable scaling to eight agents and emergent asymmetric workload/selective idleness, outperforming baselines that collapse beyond 3–4 agents.
Key findings
- S-MADRL scales to 8 agents, with Table 1 reporting successful trips per episode of 46, 87, 102, 115, 114, 105, 109, and 110 for 1–8 agents. Baselines degrade sharply: MADDPG gives 33, 33, 17, 28, 3, 3, 1, 1; MAPPO gives 43, 99, 92, 57, 21, 22, 3, 2; MA-DQN and MA-A2C collapse to near-zero beyond 3–4 agents.
- Stigmergy alone improves coordination for 3 and 4 agents but struggles at 5; adding curriculum learning improves convergence for 4- and 5-agent scenarios and enables training up to 8 agents. The introduction states S-MADRL learns decentralized coordination for up to five agents in deterministic environments and up to eight agents in stochastic environments.
- Emergent strategies include asymmetric/unequal workload distribution and selective idleness: Lorenz curves show equal workload for 3 agents, increasing inequality for 4 and 5 agents, with greater divergence under IQL+GSC. The paper also identifies a Bucket-Brigade strategy, with one lane for unladen robots entering and one for laden robots returning, as common in independent learners and reinforced by S-MADRL.
Methods
- The collective excavation task is modeled as a Dec-POMDP with independent DQN learners. Agents use local 8-dimensional observations augmented with virtual pheromone information from neighboring cells; actions are North, South, West, East, and Stop. Reward shaping includes distance (+2.5), collision (−2.0), pellet pickup (+50), and successful trip (+50), with weights 0.2, 0.2, 0.2, and 0.4.
- Stigmergy is implemented as a virtual pheromone map updated by agent movement, with occupancy initialized to ρ0 and decaying as ρ(t+1) = (1−α)ρ(t) + β. Curriculum learning starts with two agents, then adds new agents sequentially while old agents’ policies remain fixed.
- Experiments use OpenAI Gym in a narrow 2D gridworld with 1–8 homogeneous agents. Training uses Double Q-learning, experience replay, three hidden layers of 128 neurons, ε-greedy exploration decaying from 100% to 2% over the first 10% of training, learning rate 10−4, discount factor 0.99, batch size 64, and Adam. Convergence typically begins after around 200 episodes of 5000 time steps; training one agent from scratch takes approximately twelve hours.
Limitations
- Stated: the approach assumes accurate sensing of pheromone intensities, which may be affected by noise in real-world systems, and it is restricted to homogeneous agents. Future work is proposed for heterogeneous roles and for refining curriculum learning to reduce its linear dependence on team size.
- Evident: the reported results are simulation-only; no physical robot deployment or hardware validation is presented.
- Evident: training cost is high—about twelve hours per agent from scratch—and training time scales linearly with the number of concurrent agents, though curriculum learning reduces this by adding agents sequentially.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
2 Valid · 5 Not valid · 4 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“One-at-a-Time (OAT) strategy [5,6]: This is more frequently observed in joint-action learners (JALs) such as MA-DQN, MADDPG, and MAPPO, where only two agents operate at a time while others wait.”
§ Deep Reinforcement Learning for Multi-Agent Coordination > 7 Discussion
What the sources say
- Partly supportedcites [5] Avinery R, Aina KO, Dyson CJ, Kuan H, Betterton MD , Goodisman MAD, Goldman DI (2023) Agitated ants: regulation and self-organization of incipient nest excavation via collisional cues. J R Soc Interface 20 sourceExperimental — accuracy not yet measured
Section 3.4 identifies a one-at-a-time (OAT) model, so the citation supports the name. But its model concerns a single ant excavating a tunnel, not two agents operating while others wait; our copy does not connect OAT to joint-action learners or MA-DQN, MADDPG, and MAPPO. (LLM, conf=0.89)
- Could not verifycites [6] Aina KO, Avinery R, Kuan HS, Betterton MD, Goodisman MA, Goldman DI (2022) Toward task capable active matter: learning to avoid clogging in confined collectives via collisions. Front Phys 9(10):735667 sourceExperimental — accuracy not yet measured
The Experiment and Discussion sections describe an adaptive rule under which some robots rest while others dig, but they do not identify a One-at-a-Time strategy or say that exactly two agents operate while the others wait. Searches for OAT, joint-action learners, the named algorithms, and the two-agent description found no passage addressing those claims. (LLM, conf=0.76)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast, state-of-the-art algorithms such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) fail to scale beyond two agents.”
§ Deep Reinforcement Learning for Multi-Agent Coordination > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indirect communication channel encodes recent occupancy and agent activity, enabling decentralized coordination without explicit message passing.”
§ Deep Reinforcement Learning for Multi-Agent Coordination > 8 Conclusion and Future Work
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is effective to avoid getting stuck in three to four agent scenarios.”
§ Deep Reinforcement Learning for Multi-Agent Coordination > 6 Simulation Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 1 and Figure 8, conventional approaches such as I-DQN, I-A2C, MA-DQN, MA-A2C, MADDPG, and MAPPO degrade rapidly as the number of agents increases, often failing to converge beyond 3-4 agents.”
- Also stated in: § 8 Conclusion and Future Work · plain
“Baseline methods achieve moderate success with 1-3 agents but collapse beyond 4 agents due to high dimensional state and action spaces.”
§ Deep Reinforcement Learning for Multi-Agent Coordination > 7 Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 8 Conclusion and Future Work · plain
29 more claims
Create a free account to see all 81 claims and every source verdict.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 28
- The quote only partly states this claim. 3
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 2×
“The recent advancement of deep reinforcement learning (DRL) in tackling real-world tasks has established it as a powerful tool for solving complex problems characterized by high-dimensional information representation through trial-and-error methods [11].”
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Ant colony system: a cooperative learning approach to the traveling salesman problem
Dorigo, M., & Gambardella, L. M. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585892 · Cited by 8,081 · cited 1×
“Due to its scalability, robustness, low computational overhead, and adaptability [20], stigmergy has been widely adopted in multi-agent coordination and control.”
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 1×
“Its extension to multi-agent systems (MAS) has gained substantial traction in recent years, enabling innovative collaborative and distributed solutions to challenges that traditional methods struggle to address [13].”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017) · arXiv · doi:10.48550/arxiv.1706.02275 · Cited by 3,377
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · cited 1×
“By leveraging environmental memory via stigmergy, agents coordinate indirectly, mitigating non-stationarity without explicit communication or centralized critics.”
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 4×
“However, these methods suffer from significant communication overhead, and the curse of dimensionality respectively [14,17].”
InfluentialLearning to Communicate with Deep Multi-Agent Reinforcement Learning
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06676 · Cited by 1,217 · cited 1×
“…have been proposed to address this issue: 1) team rewards which encourages cooperative behavior by providing agents with shared rewards that align their objectives, and 2) Direct Communication Channels allowing agents to exchange information to reduce uncertainty and improve coordination [28].”
Deep Reinforcement Learning framework for Autonomous Driving
Sallab, A. E., Abdou, M., Perot, E., & Yogamani, S. (2017) · Electronic Imaging · doi:10.2352/issn.2470-1173.2017.19.avm-023 · Cited by 1,045 · cited 1×
“DRL has shown success in diverse domains including computer games [8], autonomous driving [9], and robotic manipulation [10].”
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957 · cited 1×
“To address these issues and enable convergence and scalability, prior research has explored several approaches, such as incorporating explicit communication channels between agents, training with joint action and full state information, and directly modeling the behavior of other agents [14–16].”
Research works printed but not linked: 1
- Holland OE (1996) Multiagent systems: lessons from social insects and collective robotics. In: Adaptation, coevolution and learning in multiagent systems: papers from the 1996 AAAI Spring Symposium, pp 57–62
Cited by
Federated swarm-coordinated deep reinforcement learning for distributed decision-making systems
Haddad, T. A. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01217-z
“For example, stigmer-gic communication and indirect coordination mechanisms have been introduced to enable robust decentralized decision-making without explicit message passing [20].”
Biological Basis of Stigmergy as an Inspiration for Modeling the Behavior of Swarm Robots
Kot, A., & Nawrocka, A. (2026) · International Carpathian Control Conference · doi:10.1109/iccc71363.2026.11593189
“This process is crucial because it ensures the system's adaptability and flexibility [10], [11], [12].”
An Individual Approach to Robot Swarm Modelling
Rączka, W., & Marszalik, D. (2026) · International Carpathian Control Conference · doi:10.1109/iccc71363.2026.11593177
“An interesting approach is the introduction of the concept of Swarm Intelligence (AI) [12], which represents a paradigm shift in artificial intelligence, using collective swarms and their behaviors of decentralized, self-organizing systems to solve given problems [13], [14], [15], [16].”
Closed-loop agentic AI in drug discovery
Chhetri, P. (2026) · RSC Medicinal Chemistry · doi:10.1039/d6md00319b · Cited by 4
Research on Multi-Agent Collaborative Task Allocation System in Cross-Border E-Commerce Scenarios
Wang, W., Yin, W., Li, S., et al. (2026) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase70763.2026.11619388
“…assigns a unique SubTask ID to each subtask and maintains a Task Decomposition Table containing: Current status of each subtask (pending, processing, completed, failed) The communication module (WebSocket-based) tracks the real-time status of each subtask with a feedback cycle of 20 seconds [9].”
Metadata
- Authors
- Aina, K. O., & Ha, S.
- Affiliations
- Georgia Institute of Technology (US)
- Corresponding author
- Ha, Sehoon
- ORCID
- Ha, Sehoon
- Venue
- Artificial Life and Robotics
- Published
- 2025-11-26
- Volume
- 31
- Issue
- 2
- Pages
- 484-494
- ISSN
- 1433-5298 (ISSN-L)
- 1614-7456
- OpenAlex
- W4416714443
- Semantic Scholar
- 56ebca1a04b3c2609c9902971633dfd4155de55b
- Cited by
- 8
- References
- 24
- Citations by year
- 2026: 8
- FWCI
- 3.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Artificial Life and Robotics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Georgia Institute of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Mechanical Engineering › Modular Robots and Swarm Intelligence
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Keywords
- multi-agent coordination, deep reinforcement learning, stigmergy, virtual pheromones, distributed coordination, curriculum learning, multi-robot systems, emergent behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep reinforcement learning for multi-agent coordination. https://app.cochat.ai/papers/10.1007/s10015-025-01089-z
- The paper
- Aina, K. O., & Ha, S. (2026). Deep reinforcement learning for multi-agent coordination. Artificial Life and Robotics. https://doi.org/10.1007/s10015-025-01089-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.