Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.

Mutual influence between language and perception in multi-agent communication games

Open access
Ohmer, Xenia, Marino, Michael, Franke, Michael, König, Peter · 2022 · PLOS Computational Biology

Abstract

Language interfaces with many other cognitive domains. This paper explores how interactions at these interfaces can be studied with deep learning methods, focusing on the relation between language emergence and visual perception. To model the emergence of language, a sender and a receiver agent are trained on a reference game. The agents are implemented as deep neural networks, with dedicated vision and language modules. Motivated by the mutual influence between language and perception in cognition, we apply systematic manipulations to the agents’ (i) visual representations, to analyze the effects on emergent communication, and (ii) communication protocols, to analyze the effects on visual representations. Our analyses show that perceptual biases shape semantic categorization and communicative content. Conversely, if the communication protocol partitions object space along certain attributes, agents learn to represent visual information about these attributes more accurately, and the representations of communication partners align. Finally, an evolutionary analysis suggests that visual representations may be shaped in part to facilitate the communication of environmentally relevant distinctions. Aside from accounting for co-adaptation effects between language and perception, our results point out ways to modulate and improve visual representation learning and emergent communication in artificial agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pcbi.1010658

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Schematic visualization of sender and receiver architecture and their interaction in one round of the reference game. The sender takes an image of the target object as input. The image is processed by the sender’s vision module and the resulting activations are used to initialize the hidden state, h0, of the sender’s language module. The initial input to the sender’s language module, 〈start〉, is a zero vector of the same dimensionality as the symbol embeddings, and at each time step a symbol is sampled from its output distribution. The generated message is processed by the receiver’s language module. In addition, the target and the distractor images are processed by the receiver’s vision module. The final selection probability is proportional to the dot product between the receiver’s final hidden state and the image embeddings.
Fig 1 Schematic visualization of sender and receiver architecture and their interaction in one round of the reference game. The sender takes an image of the target object as input. The image is processed by the sender’s vision module and the resulting activations are used to initialize the hidden state, h0, of the sender’s language module. The initial input to the sender’s language module, 〈start〉, is a zero vector of the same dimensionality as the symbol embeddings, and at each time step a symbol is sampled from its output distribution. The generated message is processed by the receiver’s language module. In addition, the target and the distractor images are processed by the receiver’s vision module. The final selection probability is proportional to the dot product between the receiver’s final hidden state and the image embeddings.CC BY · PubMed Central
Fig 2 Creating perceptual bias with relational label smoothing. (A) Example of how the training targets (labels) are adapted to induce a color bias. To generate a CNN with a color bias, some of the target weight is spread across all other classes that have the same color as the target object. In our data set, there are 64 different object classes. The first sixteen classes comprise red objects (classes 1–16), followed by yellow objects (classes 17–32), turquoise objects (class 33–48), and purple objects (classes 49–64). For example, if the input image belongs to class 2 (“tiny red cylinder”), the usual target label, y0, is a one-hot vector where the entire weight lies on the true class index. The relational component, yr, spreads the target weight onto all other red objects. The target vector used for training is a weighted average of the original target and the relational component. Analogously, to introduce a scale/shape bias, some of the target weight is spread onto all other objects of the same scale/shape as the input object. (B) Representational similarity matrix for the color CNN after training (σ = 0.6). Entries at position (i,j) correspond to the average cosine similarity between the CNN activations for images of class i and the CNN activations for images of class j (based on the penultimate fully-connected layer). The white 16 × 16 blocks on the diagonal indicate that objects of the same color are perceived as very similar to each other.
Fig 2 Creating perceptual bias with relational label smoothing. (A) Example of how the training targets (labels) are adapted to induce a color bias. To generate a CNN with a color bias, some of the target weight is spread across all other classes that have the same color as the target object. In our data set, there are 64 different object classes. The first sixteen classes comprise red objects (classes 1–16), followed by yellow objects (classes 17–32), turquoise objects (class 33–48), and purple objects (classes 49–64). For example, if the input image belongs to class 2 (“tiny red cylinder”), the usual target label, y0, is a one-hot vector where the entire weight lies on the true class index. The relational component, yr, spreads the target weight onto all other red objects. The target vector used for training is a weighted average of the original target and the relational component. Analogously, to introduce a scale/shape bias, some of the target weight is spread onto all other objects of the same scale/shape as the input object. (B) Representational similarity matrix for the color CNN after training (σ = 0.6). Entries at position (i,j) correspond to the average cosine similarity between the CNN activations for images of class i and the CNN activations for images of class j (based on the penultimate fully-connected layer). The white 16 × 16 blocks on the diagonal indicate that objects of the same color are perceived as very similar to each other.CC BY · PubMed Central
Fig 3 Illustration of the training setups. The vision module is represented by an eye, the language module by a mouth (sender) or an ear (receiver). The speech bubble represents the message, and the question mark the receiver’s selection. Modules that are not trained, i.e. have fixed weights, are light gray. Modules that are trained are dark gray. Note that the vision modules in the two language emergence scenarios (center and bottom row) are trained on the communication game and simultaneously also on the original object classification task.
Fig 3 Illustration of the training setups. The vision module is represented by an eye, the language module by a mouth (sender) or an ear (receiver). The speech bubble represents the message, and the question mark the receiver’s selection. Modules that are not trained, i.e. have fixed weights, are light gray. Modules that are trained are dark gray. Note that the vision modules in the two language emergence scenarios (center and bottom row) are trained on the communication game and simultaneously also on the original object classification task.CC BY · PubMed Central
Fig 4 Quantifying perceptual bias. (A) Object similarities calculated from 3-hot encodings based on all three attributes. This template is used in the RSA calculation to measure how well conceptually relevant attributes are encoded. (B) Object similarities calculated from 1-hot encodings based on color value. This template is used to calculate RSAcolor.
Fig 4 Quantifying perceptual bias. (A) Object similarities calculated from 3-hot encodings based on all three attributes. This template is used in the RSA calculation to measure how well conceptually relevant attributes are encoded. (B) Object similarities calculated from 1-hot encodings based on color value. This template is used to calculate RSAcolor.CC BY · PubMed Central
Fig 5 Schema of the information in the target objects, O, the corresponding messages, M, and objects selected by the receiver, S. H denotes entropy and I mutual information. The object-selection interface is entirely predicted by the messages as the mutual information between objects and selections given messages (shaded region on the left side) is zero. Therefore we can separate the analysis of sender (objects-messages) and receiver (messages-selections) as shown on the right. Note, the schema is not an actual set-theoretic representation and serves illustrative purposes only.
Fig 5 Schema of the information in the target objects, O, the corresponding messages, M, and objects selected by the receiver, S. H denotes entropy and I mutual information. The object-selection interface is entirely predicted by the messages as the mutual information between objects and selections given messages (shaded region on the left side) is zero. Therefore we can separate the analysis of sender (objects-messages) and receiver (messages-selections) as shown on the right. Note, the schema is not an actual set-theoretic representation and serves illustrative purposes only.CC BY · PubMed Central
Fig 6 Effectiveness per attribute for different pairings of senders and receivers. Pairings are (A) biased sender and biased receiver, (B) biased sender and default receiver, and (C) default sender and biased receiver. The x-axis shows the agents’ perceptual biases. The bars are labeled with the attribute a used for calculating E(Oa|M), with attributes enforced via label smoothing in dark gray. We report means and bootstrapped 95% CIs of twenty runs each.
Fig 6 Effectiveness per attribute for different pairings of senders and receivers. Pairings are (A) biased sender and biased receiver, (B) biased sender and default receiver, and (C) default sender and biased receiver. The x-axis shows the agents’ perceptual biases. The bars are labeled with the attribute a used for calculating E(Oa|M), with attributes enforced via label smoothing in dark gray. We report means and bootstrapped 95% CIs of twenty runs each.CC BY · PubMed Central
Fig 7 Influence of linguistic biases on perception. Shown are the effects of language learning and language emergence on a default agent, when paired with agents of different visual bias conditions. The left column covers the language learning scenario with a default receiver, the central column the language emergence scenario with a default receiver, and the right column the language emergence scenario with a default sender. In the language learning scenario, the sender’s weights (and therefore also the language) are entirely fixed. In the language emergence scenario, both agents are trained and the language emerges. The visual bias of the communication partner is shown on the x-axis. The top row shows the RSA scores between the default agent’s visual representations and each object attribute—indicated by the bar label—after training. Attributes that were enforced to create the visual bias of the communication partner are dark gray. The bottom row shows the RSA scores between the visual representations of the default agent and those of its communication partner before (light gray) and after (dark gray) training. Reported are means and bootstrapped 95% CIs of ten runs each.
Fig 7 Influence of linguistic biases on perception. Shown are the effects of language learning and language emergence on a default agent, when paired with agents of different visual bias conditions. The left column covers the language learning scenario with a default receiver, the central column the language emergence scenario with a default receiver, and the right column the language emergence scenario with a default sender. In the language learning scenario, the sender’s weights (and therefore also the language) are entirely fixed. In the language emergence scenario, both agents are trained and the language emerges. The visual bias of the communication partner is shown on the x-axis. The top row shows the RSA scores between the default agent’s visual representations and each object attribute—indicated by the bar label—after training. Attributes that were enforced to create the visual bias of the communication partner are dark gray. The bottom row shows the RSA scores between the visual representations of the default agent and those of its communication partner before (light gray) and after (dark gray) training. Reported are means and bootstrapped 95% CIs of ten runs each.CC BY · PubMed Central
Fig 8 RSA scores between symbolic object representations (k-hot attribute vectors) and neural object representations in the agent’s vision module. Shown are the scores for the default agent after training, for different communication partners, across ten runs each. For the language learning scenario, the default receiver is shown (left). For the language emergence scenario, the default receiver (left) and the default sender (right) are shown. The dashed line indicates the RSA score of the default CNN—so the agent’s vision module—before training.
Fig 8 RSA scores between symbolic object representations (k-hot attribute vectors) and neural object representations in the agent’s vision module. Shown are the scores for the default agent after training, for different communication partners, across ten runs each. For the language learning scenario, the default receiver is shown (left). For the language emergence scenario, the default receiver (left) and the default sender (right) are shown. The dashed line indicates the RSA score of the default CNN—so the agent’s vision module—before training.CC BY · PubMed Central
Fig 9 Mean reward on the test set for two agents of different bias types communicating with each other. For each sender-receiver combination, we ran twenty simulations. To obtain the average reward for an agent of bias type t′ communicating with an agent of bias type t, we average the rewards of the combinations t′-sender/t-receiver and t-sender/t′-receiver, hence the matrices are symmetric. We highlight the results for the combinations where both agents are biased towards all relevant attributes. (A) shows the mean test rewards for agents with t′, t ∈ {default, color, scale, shape, all} in the basic reference game where all attributes (color, scale, shape) are relevant. (B) shows the mean test rewards for agents with mixed biases t′, t ∈ {color-scale, color-shape, scale-shape} for reference games where out of the three attributes either color (left), scale (center), or shape (right) is not relevant.
Fig 9 Mean reward on the test set for two agents of different bias types communicating with each other. For each sender-receiver combination, we ran twenty simulations. To obtain the average reward for an agent of bias type t′ communicating with an agent of bias type t, we average the rewards of the combinations t′-sender/t-receiver and t-sender/t′-receiver, hence the matrices are symmetric. We highlight the results for the combinations where both agents are biased towards all relevant attributes. (A) shows the mean test rewards for agents with t′, t ∈ {default, color, scale, shape, all} in the basic reference game where all attributes (color, scale, shape) are relevant. (B) shows the mean test rewards for agents with mixed biases t′, t ∈ {color-scale, color-shape, scale-shape} for reference games where out of the three attributes either color (left), scale (center), or shape (right) is not relevant.CC BY · PubMed Central
Fig 10 Example inputs if object color is irrelevant in the communication game. The receiver target is marked by a black box. S4 Fig shows examples of sender and receiver inputs for each game variant (color irrelevant, scale irrelevant, shape irrelevant).
Fig 10 Example inputs if object color is irrelevant in the communication game. The receiver target is marked by a black box. S4 Fig shows examples of sender and receiver inputs for each game variant (color irrelevant, scale irrelevant, shape irrelevant).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 75 references

Research works printed but not linked: 17

  1. D Lewis, 1969, Convention
  2. HH Clark, 1992, Arenas of language use
  3. Ohmer X, König P, Franke M. Reinforcement of semantic representations in pragmatic agents leads to the emergence of a mutual exclusivity bias. In: Proceedings of the 42nd Annual Meeting of the Cognitive Science Society (CogSci); 2020. p. 1779–1785.
  4. Choi E, Lazaridou A, de Freitas N. Compositional obverter communication learning from raw visual input. In: Proceedings of the 6th International Conference on Learning Representations (ICLR); 2018. p. 1–18.
  5. Li F, Bowling M. Ease-of-teaching and language structure from emergent communication. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS); 2019. p. 1–11.
  6. Ren Y, Guo S, Labeau M, Cohen SB, Kirby S. Compositional languages emerge in a neural iterated learning model. In: Proceedings of the 8th International Conference on Learning Representations (ICLR); 2020. p. 1–22.
  7. G Lakoff, 1980, Metaphors we live by
  8. Havrylov S, Titov I. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS); 2017. p. 2149–2159.
  9. Mikolov T, Joulin A, Baroni M. A Roadmap towards machine intelligence. arXiv preprint. 2015;arXiv:1511.08130. · arXiv
  10. Bleys J, Loetzsch M, Spranger M, Steels L. The grounded colour naming game. In: Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (Ro-Man); 2009. p. 1–7.
  11. Lazaridou A, Peysakhovich A, Baroni M. Multi-agent cooperation and the emergence of (natural) language. In: Proceedings of the 5th International Conference on Learning Representations (ICLR); 2017. p. 1–11.
  12. Hill F, Clark S, Hermann KM, Blunsom P. Simulating early word learning in situated connectionist agents. In: Proceedings of the 42nd Annual Meeting of the Cognitive Science Society (CogSci); 2020. p. 875–881.
  13. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL); 2019. p. 4171–4186.
  14. Gandhi K, Lake BM. Mutual exclusivity as a challenge for neural networks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS); 2020. p. 14182–14192.
  15. Harnad S, Hanson SJ, Lubin J. Categorical perception and the evolution of supervised learning in neural nets. In: Powers DW, Reeker L, editors. Working Papers of the AAAI Spring Symposium on Machine Learning of Natural Language and Ontology; 1991. p. 65–74.
  16. A Noë, 2004, Action in perception
  17. KR Storrs, 2021, Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting, Journal of Cognitive Neuroscience, 33, 2044

Cited by

See all 7

Metadata

Authors
Ohmer, X., Marino, M., Franke, M., & König, P.
Affiliations
Osnabrück University (DE)
Corresponding author
Ohmer, Xenia
Venue
PLOS Computational Biology
Published
2022-10-31
Volume
18
Issue
10
Pages
e1010658
ISSN
  • 1553-734X (ISSN-L)
  • 1553-7358
OpenAlex
W4307698991
Cited by
6
References
75
Citations by year
2026: 1 · 2025: 3 · 2024: 1 · 2023: 1
FWCI
6.32
Type
journal article
Language
en
Open access
Locations
  • PLoS Computational Biology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Deutsche Forschungsgemeinschaft (GRK 2340)
Topics
  • Social Sciences › Cultural Studies › Language and cultural evolution
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
  • Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
Keywords
language emergence, visual perception, multi-agent communication, communication games, visual representation learning, communication protocols, semantic classification, perceptual bias, co-adaptation, evolutionary analysis, deep neural network
MeSH terms
Cognition, Communication, Language, Semantics, Visual Perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Mutual influence between language and perception in multi-agent communication games. https://app.cochat.ai/papers/10.1371/journal.pcbi.1010658
The paper
Ohmer, X., Marino, M., Franke, M., & König, P. (2022). Mutual influence between language and perception in multi-agent communication games. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1010658

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.