CoChat
Sign Up
PreprintCited by 195Reinforcement Learning in RoboticsLanguage and cultural evolutionNatural Language Processing Techniques

Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

Open access
Havrylov, Serhii, Titov, Ivan·2017·arXiv
DOI 10.48550/arxiv.1705.11192arXiv 1705.11192
Open this paper in CoChat — free account

Abstract

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1705.11192

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Covert communication channels, including emergent steganographic protocols, can facilitate hidden data exfiltration or coordinated leakage across agents, further complicating detection and attribution.

    “Moreover, covert communication channels, including emergent steganographic protocols, can facilitate hidden data exfiltration or coordinated leakage across agents, further complicating detection and attribution (Lazaridou et al., 2016;Havrylov & Titov, 2017;Motwani et al., 2024d).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Havrylov & Titov, 2017

    The Introduction (p. 1) reports that agents develop a communication protocol, and Section 2.2 (p. 3) notes that its words can have meanings different from those in natural language. These passages support emergent inter-agent communication, but the paper does not describe steganographic channels, data exfiltration, coordinated leakage, or difficulties with detection and attribution. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. David Barber and Felix V Agakov. The IM Algorithm: A Variational Approach to Information Maximization. In Advances in Neural Information Processing Systems , 2003.
  2. Andrea Baronchelli, Maddalena Felici, Vittorio Loreto, Emanuele Caglioti, and Luc Steels. Sharp transition towards shared vocabularies in multi-agent systems. Journal of Statistical Mechanics: Theory and Experiment , 2006(06):P06014, 2006.
  3. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. · arXiv
  4. Shalabh Bhatnagar, HL Prasad, and LA Prashanth. Stochastic recursive algorithms for optimization: simultaneous perturbation methods , volume 434. Springer, 2012.
  5. Henry Brighton. Compositional syntax from cultural transmission. Artificial life , 8(1):25-54, 2002.
  6. Donna Byron, Alexander Koller, Kristina Striegnitz, Justine Cassell, Robert Dale, Johanna Moore, and Jon Oberlander. Report on the first NLG challenge on generating instructions in virtual environments (GIVE). In Proceedings of the 12th European workshop on natural language generation , 2009.
  7. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015. · arXiv
  8. Grzegorz Chrupała, Akos Kádár, and Afra Alishahi. Learning language through pictures. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics , 2015.
  9. Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning. In Proceedings of International Conference on Computer Vision and Image Processing , 2017.
  10. Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , pages 2137-2145, 2016.
  11. Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 conference on empirical methods in natural language processing , pages 410-419. Association for Computational Linguistics, 2010.
  12. Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory Augmented Neural Networks with Wormhole Connections. arXiv preprint arXiv:1701.08718 , 2017. · arXiv
  13. James Henderson, Oliver Lemon, and Kallirroi Georgila. Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets. Computational Linguistics , 34(4):487-511, 2008.
  14. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.
  15. Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In Proceedings of the International Conference on Learning Representations , 2017.
  16. Emilio Jorge, Mikael Kågebäck, and Emil Gustavsson. Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence. In Neural Information Processing Systems, the 3rd Deep Reinforcement Learning Workshop , 2016.
  17. Diederik P Kingma and Max Welling. Auto-encoding Variational Bayes. In Proceedings of the 3rd International Conference for Learning Representations , 2014.
  18. Tomáš Koˇ cisk` y, Gábor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. Semantic parsing with semi-supervised sequential autoencoders. arXiv preprint arXiv:1609.09315 , 2016. · arXiv
  19. Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In Proceedings of the International Conference on Learning Representations , 2017.
  20. Oliver Lemon, Alexander Gruenstein, and Stanley Peters. Collaborative activities and multi-tasking in dialogue systems: Towards natural dialogue with robots. TAL. Traitement automatique des langues , 43(2):131-154, 2002.
  21. Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In Proceedings of the International Conference on Learning Representations , 2017.
  22. Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2016.
  23. Tomas Mikolov, Armand Joulin, and Marco Baroni. A roadmap towards machine intelligence. In Neural Information Processing Systems, Reasoning, Attention, and Memory Workshop , 2015.
  24. Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on Machine Learning , 2014.
  25. Stefano Nolfi and Marco Mirolli. Evolution of communication and language in embodied agents . Springer Science & Business Media, 2009.
  26. M. A. Nowak and D. Krakauer. The evolution of language. PNAS , 96(14):8028-8033, 1999. doi: 10.1073/pnas.96.14.8028. URL http://groups.lis.illinois.edu/amag/langev/paper/ nowak99theEvolution.html . · DOI
  27. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review , 21(2):97-126, 2006.
  28. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations , 2015.
  29. Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning Multiagent Communication with Backpropagation. In Advances in Neural Information Processing Systems , pages 2244-2252, 2016.
  30. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3156-3164, 2015.
  31. Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior , 11(1):37-69, 2003.
  32. M. Werning, W. Hinzen, and M. Machery. The Oxford handbook of compositionality . Oxford, UK, 2011.
  33. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992.

Cited by

Sorted by citations

Authors
Havrylov, S., & Titov, I.
Affiliations
University of Amsterdam (NL)
ORCID
Titov, Ivan
Venue
arXiv
Publisher
arXiv
Published
2017
DOI
10.48550/arxiv.1705.11192
arXiv
1705.11192
OpenAlex
W2621379712
Cited by
195
References
2
Citations by year
2025: 8 · 2024: 15 · 2023: 14 · 2022: 13 · 2021: 44 · 2020: 44 · 2019: 35 · 2018: 13 · 2017: 9
FWCI
12.33
Type
conference paper
Language
en
Open access
Licence
Locations
  • UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (639.022.518)
  • H2020 European Research Council (678254)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Cultural Studies › Language and cultural evolution
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
language emergence, multi-agent games, referential games, communication protocols, reinforcement learning, compositionality, linguistic variability, prior knowledge incorporation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review