CoChat

Deep reinforcement learning for multi-agent coordination

Open access
Aina, Kehinde O., Ha, Sehoon · 2026 · Artificial Life and Robotics

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 4 of the 8 sources its claims cite were checked for support.

Abstract

We address the challenge of coordinating multiple robots in narrow and confined environments, where congestion and interference often hinder collective task performance. Drawing inspiration from insect colonies, which achieve robust coordination through stigmergy—modifying and interpreting environmental traces—we propose a stigmergic multi-agent deep reinforcement learning (S-MADRL) framework that leverages virtual pheromones to model local and social interactions, enabling decentralized emergent coordination without explicit communication. To overcome the convergence and scalability limitations of existing algorithms such as MADQN, MADDPG, and MAPPO, we leverage curriculum learning, which decomposes complex tasks into progressively harder sub-problems. Simulation results show that our framework achieves the most effective coordination of up to eight agents, where robots self-organize into asymmetric workload distributions that reduce congestion and modulate group performance. This emergent behavior, analogous to strategies observed in nature, demonstrates a scalable solution for decentralized multi-agent coordination under crowded environments with communication constraints.

Overview

CoChat Summary

TL;DR

The paper proposes S-MADRL, a decentralized multi-agent deep RL framework that uses virtual pheromone stigmergy and curriculum learning to coordinate homogeneous robots in a narrow pellet-retrieval task. It reports stable scaling to eight agents and emergent asymmetric workload/selective idleness, outperforming baselines that collapse beyond 3–4 agents.

Key findings

  • S-MADRL scales to 8 agents, with Table 1 reporting successful trips per episode of 46, 87, 102, 115, 114, 105, 109, and 110 for 1–8 agents. Baselines degrade sharply: MADDPG gives 33, 33, 17, 28, 3, 3, 1, 1; MAPPO gives 43, 99, 92, 57, 21, 22, 3, 2; MA-DQN and MA-A2C collapse to near-zero beyond 3–4 agents.
  • Stigmergy alone improves coordination for 3 and 4 agents but struggles at 5; adding curriculum learning improves convergence for 4- and 5-agent scenarios and enables training up to 8 agents. The introduction states S-MADRL learns decentralized coordination for up to five agents in deterministic environments and up to eight agents in stochastic environments.
  • Emergent strategies include asymmetric/unequal workload distribution and selective idleness: Lorenz curves show equal workload for 3 agents, increasing inequality for 4 and 5 agents, with greater divergence under IQL+GSC. The paper also identifies a Bucket-Brigade strategy, with one lane for unladen robots entering and one for laden robots returning, as common in independent learners and reinforced by S-MADRL.

Methods

  • The collective excavation task is modeled as a Dec-POMDP with independent DQN learners. Agents use local 8-dimensional observations augmented with virtual pheromone information from neighboring cells; actions are North, South, West, East, and Stop. Reward shaping includes distance (+2.5), collision (−2.0), pellet pickup (+50), and successful trip (+50), with weights 0.2, 0.2, 0.2, and 0.4.
  • Stigmergy is implemented as a virtual pheromone map updated by agent movement, with occupancy initialized to ρ0 and decaying as ρ(t+1) = (1−α)ρ(t) + β. Curriculum learning starts with two agents, then adds new agents sequentially while old agents’ policies remain fixed.
  • Experiments use OpenAI Gym in a narrow 2D gridworld with 1–8 homogeneous agents. Training uses Double Q-learning, experience replay, three hidden layers of 128 neurons, ε-greedy exploration decaying from 100% to 2% over the first 10% of training, learning rate 10−4, discount factor 0.99, batch size 64, and Adam. Convergence typically begins after around 200 episodes of 5000 time steps; training one agent from scratch takes approximately twelve hours.

Limitations

  • Stated: the approach assumes accurate sensing of pheromone intensities, which may be affected by noise in real-world systems, and it is restricted to homogeneous agents. Future work is proposed for heterogeneous roles and for refining curriculum learning to reduce its linear dependence on team size.
  • Evident: the reported results are simulation-only; no physical robot deployment or hardware validation is presented.
  • Evident: training cost is high—about twelve hours per agent from scratch—and training time scales linearly with the number of concurrent agents, though curriculum learning reduces this by adding agents sequentially.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10015-025-01089-z

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Experimental setup of the multi-robot excavation task. (A) Real-world top view of the excavation arena, consisting of a pellet source, a narrow tunnel, excavating robots, and a home area. (B) Corresponding abstracted simulation model used for training and evaluation. The simplified representation preserves essential components in the real-world model, and enables scalable and varied experiment scenarios.
Fig. 1 Experimental setup of the multi-robot excavation task. (A) Real-world top view of the excavation arena, consisting of a pellet source, a narrow tunnel, excavating robots, and a home area. (B) Corresponding abstracted simulation model used for training and evaluation. The simplified representation preserves essential components in the real-world model, and enables scalable and varied experiment scenarios.CC-BY · From the PDF
Fig. 2 Proposed scalable decentralized MADRL framework with stigmergic communication. Each agent i receives a local observation O i and selects an action a i based on its policy π i . The environment provides the resulting state S i and reward r i , while agents leave and sense pheromone traces ρ i that diffuse and decay over time. This indirect communication channel encodes recent occupancy and agent activity, enabling decentralized coordination without explicit message passing. Learning and execution are fully independent for each agent, ensuring scalability to large team sizes.
Fig. 2 Proposed scalable decentralized MADRL framework with stigmergic communication. Each agent i receives a local observation O i and selects an action a i based on its policy π i . The environment provides the resulting state S i and reward r i , while agents leave and sense pheromone traces ρ i that diffuse and decay over time. This indirect communication channel encodes recent occupancy and agent activity, enabling decentralized coordination without explicit message passing. Learning and execution are fully independent for each agent, ensuring scalability to large team sizes.CC-BY · From the PDF
Fig. 3 Schematic of the digital pheromone map and agents' restricted field of view. Agents deposit virtual pheromones while moving, generating spatial gradients (shown in different shades of gray) that diffuse and decay over time. Each agent perceives only a limited local region (red arrows), mimicking partial observability. The blue agent is laden with pellet and returning to the home area, while green agents are searching for pellets. Stigmergic communication provides environmental memory that supports implicit coordination among agents.
Fig. 3 Schematic of the digital pheromone map and agents' restricted field of view. Agents deposit virtual pheromones while moving, generating spatial gradients (shown in different shades of gray) that diffuse and decay over time. Each agent perceives only a limited local region (red arrows), mimicking partial observability. The blue agent is laden with pellet and returning to the home area, while green agents are searching for pellets. Stigmergic communication provides environmental memory that supports implicit coordination among agents.CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Fig. 5 Learning curve comparison of the four MADRL techniques (IQL, IQL+G, IQL+GS, and IQL+GSC) for one to five agents. Baseline methods (IQL, IQL+G) fail to converge beyond three agents. Stigmergic communication (IQL+GS) enhances learning stability, while curriculum learning (IQL+GSC) further accelerates convergence and achieves the highest rewards in four- and five-agent scenarios.
Fig. 5 Learning curve comparison of the four MADRL techniques (IQL, IQL+G, IQL+GS, and IQL+GSC) for one to five agents. Baseline methods (IQL, IQL+G) fail to converge beyond three agents. Stigmergic communication (IQL+GS) enhances learning stability, while curriculum learning (IQL+GSC) further accelerates convergence and achieves the highest rewards in four- and five-agent scenarios.CC-BY · From the PDF
Fig. 6 Lorenz curve comparison of workload distribution under stigmergy-only learning (IQL+GS, panel A ) and stigmergy with curriculum learning (IQL+GSC, panel B ). For three agents, both methods result in equal workload distribution. For four and five agents, however, workload becomes increasingly unequal, especially under curriculum learning.
Fig. 6 Lorenz curve comparison of workload distribution under stigmergy-only learning (IQL+GS, panel A ) and stigmergy with curriculum learning (IQL+GSC, panel B ). For three agents, both methods result in equal workload distribution. For four and five agents, however, workload becomes increasingly unequal, especially under curriculum learning.CC-BY · From the PDF
Fig. 7 Snapshots of various scenarios during testing. (A) Four agents often get stuck in the tunnel when trained without stigmergy. (B) Stigmergy prevents agents from getting stuck, which enables effective coordination. (C) Five agents with stigmergy can still get stuck due to the difficultly of the scenario. (D) Curriculum learning enables efficient learning in difficult scenarios, and the discovery of a bio-inspired strategy: unequal work distribution to avoid clogging.
Fig. 7 Snapshots of various scenarios during testing. (A) Four agents often get stuck in the tunnel when trained without stigmergy. (B) Stigmergy prevents agents from getting stuck, which enables effective coordination. (C) Five agents with stigmergy can still get stuck due to the difficultly of the scenario. (D) Curriculum learning enables efficient learning in difficult scenarios, and the discovery of a bio-inspired strategy: unequal work distribution to avoid clogging.CC-BY · From the PDF
Fig. 8 Learning curve comparison of our proposed S-MADRL framework against state-of-the-art multi-agent deep RL baseline algorithms for team sizes from 1 to 8 agents. Each subplot shows cumulative rewards over training episodes. Baseline methods achieve moderate success with 1-3 agents but collapse beyond 4 agents due to high dimensional state and action spaces. In contrast, S-MADRL demonstrates stable convergence and sustained high rewards even with 7-8 agents, confirming its superior scalability and coordination capabilities in crowded environments.
Fig. 8 Learning curve comparison of our proposed S-MADRL framework against state-of-the-art multi-agent deep RL baseline algorithms for team sizes from 1 to 8 agents. Each subplot shows cumulative rewards over training episodes. Baseline methods achieve moderate success with 1-3 agents but collapse beyond 4 agents due to high dimensional state and action spaces. In contrast, S-MADRL demonstrates stable convergence and sustained high rewards even with 7-8 agents, confirming its superior scalability and coordination capabilities in crowded environments.CC-BY · From the PDF

Verification

Claims the paper makes

2 Valid · 5 Not valid · 4 Not checked · 70 Own results and interpretations

29 more claims

Create a free account to see all 81 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 24
  • It may state two separate claims as one. 28
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Citations

Cites

Show all 36 references

Research works printed but not linked: 1

  1. Holland OE (1996) Multiagent systems: lessons from social insects and collective robotics. In: Adaptation, coevolution and learning in multiagent systems: papers from the 1996 AAAI Spring Symposium, pp 57–62

Cited by

See all 8

Metadata

Authors
Aina, K. O., & Ha, S.
Affiliations
Georgia Institute of Technology (US)
Corresponding author
Ha, Sehoon
Venue
Artificial Life and Robotics
Published
2025-11-26
Volume
31
Issue
2
Pages
484-494
ISSN
  • 1433-5298 (ISSN-L)
  • 1614-7456
OpenAlex
W4416714443
Cited by
8
References
24
Citations by year
2026: 8
FWCI
3.24
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Life and Robotics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Georgia Institute of Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Mechanical Engineering › Modular Robots and Swarm Intelligence
  • Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
Keywords
multi-agent coordination, deep reinforcement learning, stigmergy, virtual pheromones, distributed coordination, curriculum learning, multi-robot systems, emergent behavior

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep reinforcement learning for multi-agent coordination. https://app.cochat.ai/papers/10.1007/s10015-025-01089-z
The paper
Aina, K. O., & Ha, S. (2026). Deep reinforcement learning for multi-agent coordination. Artificial Life and Robotics. https://doi.org/10.1007/s10015-025-01089-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.