CoChat
Sign Up
PreprintCited by 13Reinforcement Learning in RoboticsMulti-Agent Systems and NegotiationGame Theory and Applications

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Emergent Discrete Communication in Semantic Spaces

Open access
Tucker, Mycal, Li, Huao, Agrawal, Siddharth, Hughes, Dana, Sycara, Katia, Lewis, Michael, Shah, Julie·2021·arXiv
DOI 10.48550/arxiv.2108.01828arXiv 2108.01828
Open this paper in CoChat — free accountOpen PDF

Abstract

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2108.01828

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two recent studies of discrete EC add corrupting noise to communication which affects communication complexity through environmental, rather than agent architecture, choices.

    “Beyond such hard-coded limits on complexity, two recent studies of discrete EC add corrupting noise to communication which affects communication complexity through environmental, rather than agent architecture, choices (Kuciński et al., 2021;Tucker et al., 2021).”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Tucker et al., 2021

    Section 3.2 models discrete communication through a channel that corrupts messages with noise, and Section 5 reports that increasing environmental noise changes how learned tokens cluster. But the paper also proposes a new agent communication architecture and compares its response to noise with one-hot agents. It does not establish that the effect on communication is due to environmental choices rather than agent architecture. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Proto. agents, introduced by Tucker et al. (2021), internally compute a onehot vector, which is then multiplied by a prototype matrix to generate discrete representations in a continuous space.

    “Proto.agents, introduced by Tucker et al. (2021), internally compute a onehot vector, which is then multiplied by a “prototype matrix” to generate discrete representations in a continuous space.”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Tucker et al. (2021)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [1][1] Felipe Almeida and Geraldo Xexéo. Word embeddings: A survey. arXiv preprint arXiv:1901.09069, 2019. · arXiv
  2. [2][2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3:1137–1155, 2003.
  3. [3][3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819–840, 2002.
  4. [4][4] Kalesha Bullard, Douwe Kiela, Joelle Pineau, and Jakob Foerster. Quasi-equivalence discovery for zero-shot emergent communication. arXiv preprint arXiv:2103.08067, 2021. · arXiv
  5. [5][5] Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pineau, and Jakob Foerster. Exploring zero-shot emergent communication in embodied multi-agent populations. arXiv preprint arXiv:2010.15896, 2020. · arXiv
  6. [6][6] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  7. [7][7] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semantics-preserving adversarial embedding networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1043–1052, 2018.
  8. [8][8] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  9. [9][9] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  10. [10][10] Matthew John Hausknecht. Cooperation and communication in multiagent deep reinforcement learning. PhD thesis, 2016.
  11. [11][11] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  12. [12][12] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In International Conference on Machine Learning, pages 4399–4410. PMLR, 2020.
  13. [17][17] H. W. Kuhn and A. W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, pages 481–492, Berkeley and Los Angeles, 1951. University of California Press.
  14. [18][18] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419, 2020. · arXiv
  15. [19][19] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th Interna- tional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  16. [20][20] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  17. [21][21] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663–7674, Online, July 2020. Association for Computational Linguistics.
  18. [22][22] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Pro- cessing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 4385–4395. Association for Computational Linguistics, 2020.
  19. [23][23] Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi- agent communication. January 2018. 6th International Conference on Learning Representations, ICLR 2018 ; Conference date: 30-04-2018 Through 03-05-2018.
  20. [24][24] Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observa- tionally augmented self-play. aaai. In ACM conference on Artificial Intelligence, Ethics, and Society, 2019.
  21. [25][25] David Lewis. Convention: A philosophical study. John Wiley & Sons, 2008.
  22. [26][26] Ryan Lowe, Abhinav Gupta, Jakob N. Foerster, Douwe Kiela, and Joelle Pineau. On the interac- tion between supervision and self-play in emergent communication. In ICLR. OpenReview.net, 2020.
  23. [27][27] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Neural Information Processing Systems (NIPS), 2017.
  24. [28][28] Qingjie Meng, Daniel Rueckert, and Bernhard Kainz. Unsupervised cross-domain image classification by distance metric guided feature alignment. In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis, pages 146–157. Springer, 2020.
  25. [29][29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  26. [30][30] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  27. [31][31] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
  28. [32][32] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. 11
  29. [33][33] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
  30. [35][35] Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. Zero-shot learning through cross-modal transfer. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, NIPS'13, page 935–943, Red Hook, NY, USA, 2013. Curran Associates Inc.
  31. [36][36] Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  32. [37][37] Mycal Tucker, Yilun Zhou, and Julie Shah. Adversarially guided self-play for adopting social conventions. arXiv preprint arXiv:2001.05994, 2020. · arXiv
  33. [38][38] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251–2265, 2018.
  34. [39][39] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403– 2412, 2018. Checklist
  35. [1]1. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Section 3.3 on learning, and Section 7 on broader limitations. (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 on how word embeddings and our technique could reinforce biases. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
  36. [5]5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix F. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] See Appendix F. (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix F. A Appendix: Optimality of One-Hot Encodings We present a brief proof about the local optimality of one-hot encodings in the decision-theoretic framework presented in Section 3.2. We seek to prove that, under assumptions of an identity reward matrix, tokens constrained to a unit hypercube, and gaussian additive noise, one-hot tokens are an optimally robust communication strategy. We only seek to prove local optimality, as one many trivially generate multiple, equally optimal tokens by, for example, flipping all bits. The following derivation uses Karush–Kuhn–Tucker (KKT) conditions, a generalization of Lagrange multipliers [17]. Theorem A.1 (Optimality of One-Hot Encodings). A matrix Tc×c, constrained to the unit hypercube, locally maximizes ∑ i∈[1,c] ∑ j∈[1,c];j 6=i(T [i]− T [j])>(T [i]− T [j]) using one-hot encodings. Proof. We maximize the function, subject to constraints. We denote the ith row of matrix T as Ti, and the jth element of the ith row as Tij . T ∗ = arg max T ∑ i∈[1,c] ∑ j∈[1,c];j 6=i (Ti − Tj)>(Ti − Tj) s.t. 0 ≤ Tij ≤ 1∀i, j ∈ [1, c] (6) We define KKT terms, using µi and λi for constraints. µij , λij ≥ 0∀i, j ∈ [1, c] (7) L = ∑ i∈[1,c] ∑ j∈[1,c];j 6=i [ (Ti − Tj)>(Ti − Tj) ] − ∑ i∈[1,c] [ ~µi(Ti −~1)− ~λiTi ]

Cited by

Sorted by citations

Metadata

Authors
Tucker, M., Li, H., Agrawal, S., Hughes, D., Sycara, K., Lewis, M., & Shah, J.
Affiliations
  • Massachusetts Institute of Technology (US)
  • University of Pittsburgh (US)
  • Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2108.01828
arXiv
2108.01828
OpenAlex
W3189582301
Cited by
13
References
2
Citations by year
2025: 2 · 2024: 5 · 2023: 4 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
emergent communication, semantic space, neural agents, reinforcement learning, word embeddings, token clustering, noisy environments, self-play

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review