CoChat
Sign Up
PreprintCited by 26Multi-Agent Systems and NegotiationReinforcement Learning in RoboticsLanguage and cultural evolution

The DOI matches its DataCite record.

Learning to Ground Multi-Agent Communication with Autoencoders

Open access
Lin, Toru, Huh, Minyoung, Stauffer, Chris, Lim, Ser-Nam, Isola, Phillip·2021·arXiv
DOI 10.48550/arxiv.2110.15349arXiv 2110.15349
Open this paper in CoChat — free accountOpen PDF

Abstract

Communication requires having a common language, a lingua franca, between agents. This language could emerge via a consensus process, but it may require many generations of trial and error. Alternatively, the lingua franca can be given by the environment, where agents ground their language in representations of the observed world. We demonstrate a simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that a standard representation learning algorithm -- autoencoding -- is sufficient for arriving at a grounded common language. When agents broadcast these representations, they learn to understand and respond to each other's utterances and achieve surprisingly strong task performance across a variety of multi-agent communication environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2110.15349

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lin et al. (2021)'s method tends to induce faster convergence, but might induce unnecessarily complex communication.

    “The authors find that their method tends to induce faster convergence (i.e., the team performs well at a task faster), but they note that their method might induce unnecessarily complex communication.”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Lin et al. (2021)

    Section 5.3 reports that the method converged to an optimal strategy after 150,000 training iterations in RedBlueDoors and completed FindGoal episodes faster than the baselines. The Discussion warns that agents communicate observed information indiscriminately and may fail when greater selectivity is needed, but it does not say that their communication is unnecessarily complex. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Lin et al. (2021) indirectly explores the role of informativeness in guiding emergent communication by using a reconstruction loss to generate communication encodings.

    “Lastly, Lin et al. (2021) indirectly explores the role of informativeness in guiding emergent communication, by using a reconstruction loss to generate communication encodings.”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Lin et al. (2021)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 45

  1. Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. 4, 14 · arXiv
  2. C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. · arXiv
  3. B. Bogin, M. Geva, and J. Berant. Emergence of communication in an interactive world with consistent speakers. arXiv preprint arXiv:1809.00549, 2018. 1, 3 · arXiv
  4. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. 13 · arXiv
  5. K. Cao, A. Lazaridou, M. Lanctot, J. Z. Leibo, K. Tuyls, and S. Clark. Emergent communication through negotiation. arXiv preprint arXiv:1804.03980, 2018. 2 · arXiv
  6. E. Choi, A. Lazaridou, and N. de Freitas. Compositional obverter communication learning from raw visual input. arXiv preprint arXiv:1804.02341, 2018. 1, 2, 3 · arXiv
  7. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. 5 · arXiv
  8. A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630, 2020. 1 · arXiv
  9. A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pages 1538-1546. PMLR, 2019. 2
  10. T. Eccles, Y. Bachrach, G. Lever, A. Lazaridou, and T. Graepel. Biases for emergent communication in multi-agent reinforcement learning. arXiv preprint arXiv:1912.05676, 2019. 1, 2, 3, 6, 7, 9 · arXiv
  11. M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pages 226-231, 1996. 8, 9
  12. K. Evtimova, A. Drozdov, D. Kiela, and K. Cho. Emergent communication in a multi-modal, multi-step referential game. arXiv preprint arXiv:1705.10369, 2017. 5 · arXiv
  13. J. Farrell. Cheap talk, coordination, and entry. The RAND Journal of Economics, pages 34-39, 1987. 6
  14. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 1, 2
  15. J. Foerster, F. Song, E. Hughes, N. Burch, I. Dunning, S. Whiteson, M. Botvinick, and M. Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 1942-1951. PMLR, 2019. 2
  16. J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson. Learning to communicate to solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv:1602.02672, 2016. 2 · arXiv
  17. J. N. Foerster, Y. M. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06676, 2016. 1, 2, 6, 7 · arXiv
  18. L. Graesser, K. Cho, and D. Kiela. Emergent linguistic phenomena in multi-agent communication games. arXiv preprint arXiv:1901.08706, 2019. 1 · arXiv
  19. S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-346, 1990. 1, 3
  20. J. R. Hurford. Biological evolution of the saussurean sign as a component of the language acquisition device. Lingua, 77(2):187-222, 1989. 2
  21. N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049. PMLR, 2019. 2
  22. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 13 · arXiv
  23. V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008-1014. Citeseer, 2000. 2
  24. A. Lazaridou, K. M. Hermann, K. Tuyls, and S. Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. arXiv preprint arXiv:1804.03984, 2018. 3, 5 · arXiv
  25. D. Lewis. Convention: A philosophical study. John Wiley & Sons, 2008. 5
  26. M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994. 3
  27. R. Lowe, J. Foerster, Y.-L. Boureau, J. Pineau, and Y. Dauphin. On the pitfalls of measuring emergent communication. arXiv preprint arXiv:1903.05168, 2019. 9 · arXiv
  28. R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275, 2017. 1, 2, 3, 7 · arXiv
  29. B. MacWhinney. The emergence of language from embodiment. Psychology Press, 2013. 1
  30. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016. 3
  31. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 2 · arXiv
  32. I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 1, 2
  33. M. Noukhovitch, T. LaCroix, A. Lazaridou, and A. Courville. Emergent communication under competition. arXiv preprint arXiv:2101.10276, 2021. 2 · arXiv
  34. M. A. Nowak and D. C. Krakauer. The evolution of language. Proceedings of the National Academy of Sciences, 96(14):8028-8033, 1999. 1
  35. D. K. Roy and A. P. Pentland. Learning words from sights and sounds: A computational model. Cognitive science, 26(1):113-146, 2002. 2
  36. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. 3 · arXiv
  37. L. S. Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953. 3
  38. L. Steels. The synthetic modeling of language origins. Evolution of communication, 1(1):1-34, 1997. 2
  39. L. Steels and F. Kaplan. Aibo's first words: The social learning of language and meaning. Evolution of communication, 4(1):3-32, 2000. 2
  40. S. Sukhbaatar, A. Szlam, and R. Fergus. Learning multiagent communication with backpropagation. arXiv preprint arXiv:1605.07736, 2016. 1, 2 · arXiv
  41. M. Tomasello. Origins of human communication. MIT press, 2010. 1
  42. T. Udagawa and A. Aizawa. A natural language corpus of common grounding under continuous and partially-observable context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7120-7127, 2019. 3
  43. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 8, 9
  44. A. Xie, D. P. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations to influence multi-agent interaction. arXiv preprint arXiv:2011.06619, 2020. 10 · arXiv
  45. L. Zintgraf, S. Devlin, K. Ciosek, S. Whiteson, and K. Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864, 2021. 10 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Lin, T., Huh, M., Stauffer, C., Lim, S.-N., & Isola, P.
Affiliations
MIT Computer Science and Artificial Intelligence Laboratory (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2110.15349
arXiv
2110.15349
OpenAlex
W3208045904
Cited by
26
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 10 · 2023: 8 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Cultural Studies › Language and cultural evolution
Keywords
multi-agent communication, autoencoder, representation learning, lingua franca, distributed coordination, multi-agent reinforcement learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review