Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 10 of the 55 sources its claims cite were checked for support.

Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour

Open access
Macmillan-Scott, Olivia, Musolesi, Mirco · 2025 · PLOS Computational Biology

Abstract

The coevolution of signalling is a complex problem within animal behaviour, and is also central to communication between artificial agents. The Sir Philip Sidney game was designed to model this dyadic interaction from an evolutionary biology perspective, and was formulated to demonstrate the emergence of honest signalling. We use Multi-Agent Reinforcement Learning (MARL) to show that in the majority of cases, the resulting behaviour adopted by agents is not that shown in the original derivation of the model. This paper demonstrates that MARL can be a powerful tool to study evolutionary dynamics and understand the underlying mechanisms of learning over generations; particularly advantageous is the interpretability of this type of approach, as well as the fact that it allows us to study emergent behaviour without the need to constrain the strategy space from the outset. Although it originally set out to exemplify honest signalling, we show that the game provides no incentive for such behaviour. In the majority of cases, the optimal outcome is one that does not require a signal for the resource to be given. This type of interaction is observed within animal behaviour and is sometimes referred to as proactive prosociality. High learning and low discount rates of the reinforcement learning model are shown to be optimal in order to achieve the outcome that maximises both agents’ reward, and proximity to the given threshold leads to suboptimal learning.

Overview

CoChat Summary

TL;DR

Using multi-agent Q-learning to replay the Sir Philip Sidney game, the authors find that agents usually do not learn Maynard Smith’s honest-signalling equilibrium; instead the Beneficiary typically never signals and the Donor either keeps or gives the resource depending on parameter thresholds. High learning and low discount rates maximize learning of the inclusive-fitness-optimal outcome, while proximity to the threshold reduces it.

Key findings

  • In the majority of cases, agents do not learn Maynard Smith’s honest-signalling strategies. The Beneficiary never has an incentive to signal; only the Donor’s behaviour changes with the threshold in Eq 1. Table 4 reports: for r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied), B learns Not signal in 83.3% of runs and D learns Keep in 97.1%; for r=0.8, V=0.1, U=0.9 (satisfied), B learns Not signal in 46.1% and D learns Give in 94.6%; for r=0.9, V=0.75, U=0.95 (satisfied), B learns Not signal in 82.4% and D learns Give in 48.2%.
  • The optimal reinforcement-learning parameters are a learning rate of 0.9 and discount rate of 0.1, which produce the expected strategy most often. For Player D in Case 1, the optimal strategy (always keep) is learned in at least 80.9% of iterations; the lowest value, 80.9%, occurs with learning rate 0.1 and discount rate 0.9. For each learning rate, a lower discount rate leads to the optimal strategy being learned a higher proportion of the time.
  • Proximity to the threshold produces suboptimal learning. D learns Always Give in Case 2 almost twice as often as in Case 3; the average Q-value difference between Give and Keep is 0.410 for Case 2 and 0.139 for Case 3. In Case 3, D learns Always Give in 48.2% of iterations, and the next most learned strategy is alternating Give/Keep in 38.5%.
  • When Eq 1 is satisfied, the emergent {Not signal, Give} outcome is assimilated to proactive prosociality; when it is not satisfied, the outcome is {Not signal, Keep}. Strong signalling is rarely learned by the Beneficiary.

Methods

  • The authors model the Sir Philip Sidney game with two agents: a Beneficiary (B) who may be thirsty with probability p or not thirsty, and a Donor (D) who holds an indivisible resource. B’s actions in the simulation are Not signal, Weak signal (c=0.25), and Strong signal (c=0.75); D’s actions are Give and Keep. Payoffs use inclusive fitness, F_i = P_i + rP_j, and the threshold condition is Eq 1: r > (1−U)/(1−V+pV).
  • Agents learn via Q-learning. Each dyad plays 500 rounds; the first 50 rounds are exploratory (ε=1), after which exploration decays as ε=2/N. Q-value tables are initialized at 0, and each parameter set is run for 1000 iterations with averages calculated. Learning rates tested are 0.1, 0.5, and 0.9; discount rates are 0.1 and 0.9. Results are checked for p=0 and p=1.
  • The analysis compares learned strategies against theoretically derived evolutionarily stable strategies, reports the proportion of runs in which each strategy is learned, and inspects Q-values over rounds. Three main parameter cases are used: r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied); r=0.8, V=0.1, U=0.9 (satisfied); and r=0.9, V=0.75, U=0.95 (satisfied, close to threshold).

Limitations

  • Stated: The study focuses on dyadic interactions, leaving open questions about population dynamics, partner selection, and learning over generations with agents initialized from a “parent” agent.
  • Stated: Learning and discount rates require interpretation when applied to animal behaviour, whereas the relatedness coefficient and signal costs can be directly understood. The length of the exploration period may also affect results; longer initial exploration or slower ε decay could allow greater separation between Q-values and increase optimal-strategy learning (S5 Appendix).
  • Evident from the text: Results are simulation-only, with no empirical animal data; the tested parameter sets and learning/discount rates are limited, and the strategy space is constrained by agent architecture, primarily memory length.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pcbi.1013302

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 Extensive-form game tree: Sir Philip Sidney game.
Fig 1 Extensive-form game tree: Sir Philip Sidney game.CC BY · PubMed Central
Fig 2 Visual representation of threshold values where the evolutionarily stable strategies presented in this article would hold (p=0.5,U=0.9,V=0.1). The intersection of r = 0.8 with both c = 0.25 and c = 0.75 is within the shaded area of the two inequalities, therefore both thresholds are satisfied and we would expect the resulting strategy combination to be {N,G}. However, for r = 0.2, the evolutionarily stable strategies of Not signal and Keep only hold for low signal costs as the intersection with c = 0.75 is outside the shaded area.
Fig 2 Visual representation of threshold values where the evolutionarily stable strategies presented in this article would hold (p=0.5,U=0.9,V=0.1). The intersection of r = 0.8 with both c = 0.25 and c = 0.75 is within the shaded area of the two inequalities, therefore both thresholds are satisfied and we would expect the resulting strategy combination to be {N,G}. However, for r = 0.2, the evolutionarily stable strategies of Not signal and Keep only hold for low signal costs as the intersection with c = 0.75 is outside the shaded area.CC BY · PubMed Central
Fig 3 Beneficiary resulting strategies. Percentage of runs that the Beneficiary learns each strategy. Not thirsty state, r=0.5,U=0.2,V=0.2. Strategies described in Table 5.
Fig 3 Beneficiary resulting strategies. Percentage of runs that the Beneficiary learns each strategy. Not thirsty state, r=0.5,U=0.2,V=0.2. Strategies described in Table 5.CC BY · PubMed Central
Fig 4 Donor resulting strategies. Percentage of runs that the Donor learns each strategy. Not thirsty state, r=0.5,U=0.2,V=0.2. Strategies described in Table 6.
Fig 4 Donor resulting strategies. Percentage of runs that the Donor learns each strategy. Not thirsty state, r=0.5,U=0.2,V=0.2. Strategies described in Table 6.CC BY · PubMed Central
Fig 5 Q-values Case 2, darker line is average across all, faint lines are average for each state.
Fig 5 Q-values Case 2, darker line is average across all, faint lines are average for each state.CC BY · PubMed Central
Fig 6 Q-values Case 3, darker line is average across all, faint lines are average for each state.
Fig 6 Q-values Case 3, darker line is average across all, faint lines are average for each state.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 2 Not valid · 29 Not checked · 95 Own results and interpretations

39 more claims

Create a free account to see all 130 claims and to verify the 45 references still open.

See all claims — free account
  • Probably not a substantial claim. 45
  • It may state two separate claims as one. 60
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

  • Mate selection—A selection for a handicap

    Zahavi, A. (1975) · Journal of Theoretical Biology · doi:10.1016/0022-5193(75)90111-3 · Cited by 5,251 · cited 4×

    “The study of signalling within dyadic interactions, both involving conflicting or common interest, has given rise to a number of theories surrounding the emergence of honest signalling [3,4].”
    Influential
  • Biological signals as handicaps

    Grafen, A. (1990) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(05)80088-8 · Cited by 2,462 · cited 2×

    “The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
  • Reciprocal food sharing in the vampire bat

    Wilkinson, G. S. (1984) · Nature · doi:10.1038/308181a0 · Cited by 1,016 · cited 1×

    “Much of the literature has come from the field of animal behaviour, where signals are crucial to interactions such as parent-offspring begging [5], food sharing [6,7] or sexual selection [3,8].”
  • The cost of honesty

    Zahavi, A. (1977) · Journal of Theoretical Biology · doi:10.1016/0022-5193(77)90061-3 · Cited by 1,000 · cited 2×

    “The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
  • Animal Signals

    Smith, J. M., & Harper, D. (2003) · doi:10.1093/oso/9780198526841.001.0001 · Cited by 945

  • Towards Flexible Teamwork

    Tambe, M. (1997) · Journal of Artificial Intelligence Research · doi:10.1613/jair.433 · Cited by 897 · cited 9×

    “The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
    Influential
  • Learning Through Reinforcement and Replicator Dynamics

    Börgers, T., & Sarin, R. (1997) · Journal of Economic Theory · doi:10.1006/jeth.1997.2319 · Cited by 737 · cited 1×

    “The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
  • The evolutionary origin of human hyper-cooperation

    Burkart, J. M., Allon, O., Amici, F., Fichtel, C., Finkenwirth, C., Heschl, A., Huber, J., Isler, K., Kosonen, Z. K., Martins, E., et al. (2014) · Nature Communications · doi:10.1038/ncomms5747 · Cited by 396 · cited 2×

    “We observe this phenomenon both in the animal realm [53,54], and within human behaviour [55], and this behaviour may be caused by a number of mechanisms.”
  • “The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
  • How Can Evolution Learn?

    Watson, R. A., & Szathmáry, E. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.11.009 · Cited by 324 · cited 1×

    “Watson et al. [26] also discuss the equivalence of evolutionary processes and simple learning processes, highlighting the use of past experience to make decisions in both types of systems.”

Show all 49 references

Research works printed but not linked: 5

  1. Lewis DK. Convention: A Philosophical Study. Cambridge: Harvard University Press; 1969.
  2. Smit J, Santos FP. Fairness and cooperation between independent reinforcement learners through indirect reciprocity. In: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. AAMAS’24; 2024. p. 2468–70.
  3. Anastassacos N, García J, Hailes S, Musolesi M. Cooperation and reputation dynamics with reinforcement learning. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. AAMAS’21; 2021. p. 115–23.
  4. Tuyls K, Lenaerts T, Verbeeck K, Maes S, Manderick B. Towards a relation between learning agents and evolutionary dynamics. In: Proceedings of the Fourteenth Belgium-Netherlands Conference on Artificial Intelligence; 2002. p. 315–22.
  5. CJCH Watkins, 1992, Q-learning, Machine Learning., 8, 279

Cited by

See all 1

Metadata

Authors
Macmillan-Scott, O., & Musolesi, M.
Affiliations
University College London (GB)
Corresponding author
Macmillan-Scott, Olivia
Venue
PLOS Computational Biology
Published
2025-08-26
Volume
21
Issue
8
Pages
e1013302
ISSN
  • 1553-734X (ISSN-L)
  • 1553-7358
OpenAlex
W4413636935
Cited by
1
References
49
Citations by year
2026: 1
FWCI
1.73
Type
journal article
Language
en
Open access
Locations
  • PLoS Computational Biology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • UCL Discovery (University College London) (submittedVersion)
Topics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Evolutionary Psychology and Human Behavior
Keywords
honest signaling, multi-agent reinforcement learning, animal communication, evolutionary dynamics, emergent behavior, learning rate, discount rate, dyadic interaction
MeSH terms
Animals, Animal Communication, Behavior, Animal, Biological Evolution, Game Theory, Humans, Learning, Models, Biological, Reinforcement, Psychology, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour. https://app.cochat.ai/papers/10.1371/journal.pcbi.1013302
The paper
Macmillan-Scott, O., & Musolesi, M. (2025). Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1013302

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.