CoChat
Sign Up
PreprintCited by 23Language and cultural evolutionGame Theory and ApplicationsEvolutionary Algorithms and Applications

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication

Open access
Kuciński, Łukasz, Korbak, Tomasz, Kołodziej, Paweł, Miłoś, Piotr·2021·arXiv
DOI 10.48550/arxiv.2111.06464arXiv 2111.06464
Open this paper in CoChat — free accountOpen PDF

Abstract

Communication is compositional if complex signals can be represented as a combination of simpler subparts. In this paper, we theoretically show that inductive biases on both the training framework and the data are needed to develop a compositional communication. Moreover, we prove that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel. We experimentally confirm that a range of noise levels, which depends on the model and the data, indeed promotes compositionality. Finally, we provide a comprehensive study of this dependence and report results in terms of recently studied compositionality metrics: topographical similarity, conflict count, and context independence.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2111.06464

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Two recent studies of discrete EC add corrupting noise to communication which affects communication complexity through environmental, rather than agent architecture, choices.

    “Beyond such hard-coded limits on complexity, two recent studies of discrete EC add corrupting noise to communication which affects communication complexity through environmental, rather than agent architecture, choices (Kuciński et al., 2021;Tucker et al., 2021).”

    From Towards Human-Like Emergent Communication via Utility, Informativeness, and Complexity (2025) · cites this paper as Kuciński et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Barrett, J. A. and Skyrms, B. (2017). Self-assembling Games. The British Journal for the Philosophy of Science, 68(2):329-353.
  2. Batali, J. (1998). Computational simulations of the emergence of grammar. Approach to the Evolution of Language, pages 405-426.
  3. Bogin, B., Geva, M., and Berant, J. (2018). Emergence of Communication in an Interactive World with Consistent Speakers. arXiv preprint arXiv:1809.00549. · arXiv
  4. Brighton, H. and Kirby, S. (2006). Understanding Linguistic Evolution by Visualizing the Emergence of Topographic Mappings. Artificial Life, 12(2):229-242.
  5. Chaabouni, R., Kharitonov, E., Bouchacourt, D., Dupoux, E., and Baroni, M. (2020). Compositionality and Generalization In Emergent Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427-4442.
  6. Choi, E., Lazaridou, A., and de Freitas, N. (2018). Compositional Obverter Communication Learning From Raw Visual Input. In International Conference on Learning Representations.
  7. Cogswell, M., Lu, J., Lee, S., Parikh, D., and Batra, D. (2019). Emergence of Compositional Language with Deep Generational Transmission. arXiv preprint arXiv:1904.09067. · arXiv
  8. Das, A., Kottur, S., Moura, J. M., Lee, S., and Batra, D. (2017). Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE international conference on computer vision, pages 2951-2960.
  9. Fodor, J. A. and Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: a critical analysis. Cognition, 28(1-2):3-71.
  10. Foerster, J. N., Assael, Y. M., de Freitas, N., and Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. NIPS'16 Proceedings of the 30th International Conference on Neural Information Processing Systems.
  11. Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. (2020). Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269.
  12. Fudenberg, D. and Tirole, J. (1991). Game Theory. MIT Press.
  13. Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677. · arXiv
  14. Guo, S., Ren, Y., Słowik, A., and Mathewson, K. (2020). Inductive bias and language expressivity in emergent communication. arXiv preprint arXiv:2012.02875. · arXiv
  15. Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2020). A very condensed survey and critique of multiagent deep reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 2146-2148.
  16. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
  17. Jang, E., Gu, S., and Poole, B. (2017). Categorical reparametrization with gumbel-softmax. In Proceedings International Conference on Learning Representations 2017.
  18. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and De Freitas, N. (2019). Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. In International Conference on Machine Learning, pages 3040-3049.
  19. Jiang, Y., Gu, S., Murphy, K., and Finn, C. (2019). Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 32.
  20. Kaiser, Ł. and Bengio, S. (2018). Discrete autoencoders for sequence models. arXiv preprint arXiv:1801.09797. · arXiv
  21. Kim, H. and Mnih, A. (2018). Disentangling by factorising. In International Conference on Machine Learning, pages 2649-2658.
  22. Kirby, S. (2001). Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2):102110.
  23. Korbak, T., Zubek, J., Kuci´ nski, Ł., Miło´ s, P., and Raczaszek-Leonardi, J. (2019). Developmentally motivated emergence of compositional communication via template transfer. arXiv preprint arXiv:1910.06079. · arXiv
  24. Korbak, T., Zubek, J., and Raczaszek-Leonardi, J. (2020). Measuring non-trivial compositionality in emergent communication. arXiv preprint arXiv:2010.15058. · arXiv
  25. Kuci´ nski, Ł., Kołodziej, P., and Miło´ s, P. (2020). Emergence of compositional language in communication through noisy channel. ICML 2020 Workshop LaReL.
  26. Lazaridou, A., Hermann, K. M., Tuyls, K., and Clark, S. (2018). Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Representations.
  27. Lazaridou, A., Peysakhovich, A., and Baroni, M. (2017). Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations.
  28. Lee, J., Cho, K., Weston, J., and Kiela, D. (2018). Emergent translation in multi-agent communication. In International Conference on Learning Representations.
  29. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Soviet Union.
  30. Li, F. and Bowling, M. (2019). Ease-of-teaching and language structure from emergent communication. In Advances in Neural Information Processing Systems, pages 15825-15835.
  31. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨ olkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114-4124.
  32. Mordatch, I. and Abbeel, P. (2018). Emergence of grounded compositional language in multi-agent populations. In Thirty-second AAAI conference on artificial intelligence.
  33. Nowak, M. A. and Krakauer, D. C. (1999). The evolution of language. Proceedings of the National Academy of Sciences, 96(14):8028-8033.
  34. Raczaszek-Leonardi, J., Nomikou, I., Rohlfing, K. J., and Deacon, T. W. (2018). Language Development From an Ecological Perspective: Ecologically Valid Ways to Abstract Symbols. Ecological Psychology, 30(1):39-73.
  35. Ren, Y., Guo, S., Labeau, M., Cohen, S. B., and Kirby, S. (2020). Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations.
  36. Resnick, C., Gupta, A., Foerster, J., Dai, A. M., and Cho, K. (2020). Capacity, Bandwidth, and Compositionality in Emergent Language Learning. In Autonomous Agents and Multiagent Systems, pages 1125-1133.
  37. Salakhutdinov, R. and Hinton, G. (2009). Semantic hashing. International Journal of Approximate Reasoning, 50(7):969-978.
  38. Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3):379-423.
  39. Skyrms, B. (2010). Signals: evolution, learning, & information. Oxford University Press, Oxford ; New York.
  40. Słowik, A., Gupta, A., Hamilton, W. L., Jamnik, M., and Holden, S. B. (2020a). Towards graph representation learning in emergent communication. arXiv preprint arXiv:2001.09063. · arXiv
  41. Słowik, A., Gupta, A., Hamilton, W. L., Jamnik, M., Holden, S. B., and Pal, C. (2020b). Exploring structural inductive biases in emergent communication. arXiv preprint arXiv:2002.01335. · arXiv
  42. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
  43. Van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9(11).

Cited by

Sorted by citations

Metadata

Authors
Kuciński, Ł., Korbak, T., Kołodziej, P., & Miłoś, P.
Affiliations
University of Sussex (GB)
ORCID
Korbak, Tomasz
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2111.06464
arXiv
2111.06464
OpenAlex
W3212220559
Cited by
23
References
2
Citations by year
2025: 17 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Leverhulme Trust
  • Infrastruktura PL-Grid
  • Poznańskie Centrum Superkomputerowo-Sieciowe
  • Academic Computer Centre Cyfronet, AGH University of Science and Technology
Topics
  • Social Sciences › Cultural Studies › Language and cultural evolution
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
inductive bias, signaling games, noisy channels, context independence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review