CoChat
Sign Up
PreprintCited by 1,217Reinforcement Learning in RoboticsMulti-Agent Systems and NegotiationGame Theory and Applications

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Open access
Foerster, Jakob N., Assael, Yannis M., de Freitas, Nando, Whiteson, Shimon·2016·arXiv
DOI 10.48550/arxiv.1605.06676arXiv 1605.06676
Open this paper in CoChat — free accountOpen PDF

Abstract

We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1605.06676

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies in machine-learning emergent communication that focused on other reinforcement learning settings have not made direct contact with general, cognitively-motivated principles believed to underlie human language, such as IB.

    “While some studies in the machine-learning emergent communication literature have focused on other reinforcement learning settings (Foerster et al., 2016;Mordatch & Abbeel, 2018, e.g.,), they have not made direct contact with general, cognitively-motivate principles that are believed to underlie human language, such as IB.”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Foerster et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [2]V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  2. [3]J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson. Learning to communicate to solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv:1602.02672, 2016. · arXiv
  3. [4]M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML, 1993.
  4. [5]F. S. Melo, M. Spaan, and S. J. Witwicki. QueryPOMDP: POMDP-based communication in multiagent systems. In Multi-Agent Systems, pages 189–204. 2011.
  5. [7]C. Zhang and V. Lesser. Coordinating multi-agent reinforcement learning with limited communication. In AAMAS, volume 2, pages 1101–1108, 2013.
  6. [8]T. Kasai, H. Tenmoto, and A. Kamiya. Learning of communication codes in multi-agent reinforcement learning problem. In IEEE Soft Computing in Industrial Applications, pages 1–6, 2008.
  7. [9]C. L. Giles and K. C. Jim. Learning communication for multi-agent systems. In Innovative Concepts for Agent-Based Systems, pages 377–390. Springer, 2002.
  8. [10]K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015. · arXiv
  9. [11]M. Courbariaux and Y. Bengio. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830, 2016. · arXiv
  10. [12]G. Hinton and R. Salakhutdinov. Discovering binary codes for documents by learning deep generative models. Topics in Cognitive Science, 3(1):74–91, 2011.
  11. [13]R. S. Sutton and A. G. Barto. Introduction to reinforcement learning. MIT Press, 1998.
  12. [14]A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779, 2015. · arXiv
  13. [15]Y. Shoham and K. Leyton-Brown. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, New York, 2009.
  14. [16]E. Zawadzki, A. Lipson, and K. Leyton-Brown. Empirically evaluating multiagent learning algorithms. arXiv preprint 1401.8074, 2014. · arXiv
  15. [17]M. Hausknecht and P. Stone. Deep recurrent Q-learning for partially observable MDPs. arXiv preprint arXiv:1507.06527, 2015. · arXiv
  16. [19]K. Narasimhan, T. Kulkarni, and R. Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941, 2015. · arXiv
  17. [20]K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014. · arXiv
  18. [22]J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. · arXiv
  19. [23]R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In ICML, pages 2342–2350, 2015.
  20. [24]S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448–456, 2015.
  21. [26]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  22. [27]M. Studdert-Kennedy. How did language go discrete? In M. Tallerman, editor, Language Origins: Perspectives on Evolution, chapter 3. Oxford University Press, 2005.

Cited by

Sorted by citations

Authors
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S.
Affiliations
University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
Volume
29
Pages
2137-2145
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1605.06676
arXiv
1605.06676
OpenAlex
W2395575420
Cited by
1,217
References
2
Citations by year
2026: 12 · 2025: 61 · 2024: 139 · 2023: 153 · 2022: 164 · 2021: 197 · 2020: 222 · 2019: 159 · 2018: 78 · 2017: 25 · 2016: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Oxford University Research Archive (ORA) (University of Oxford) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • DeepMind
  • Engineering and Physical Sciences Research Council
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
multi-agent reinforcement learning, agent communication, deep Q-learning, partial observability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review