The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 10 of the 55 sources its claims cite were checked for support.
Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour
Open accessAbstract
The coevolution of signalling is a complex problem within animal behaviour, and is also central to communication between artificial agents. The Sir Philip Sidney game was designed to model this dyadic interaction from an evolutionary biology perspective, and was formulated to demonstrate the emergence of honest signalling. We use Multi-Agent Reinforcement Learning (MARL) to show that in the majority of cases, the resulting behaviour adopted by agents is not that shown in the original derivation of the model. This paper demonstrates that MARL can be a powerful tool to study evolutionary dynamics and understand the underlying mechanisms of learning over generations; particularly advantageous is the interpretability of this type of approach, as well as the fact that it allows us to study emergent behaviour without the need to constrain the strategy space from the outset. Although it originally set out to exemplify honest signalling, we show that the game provides no incentive for such behaviour. In the majority of cases, the optimal outcome is one that does not require a signal for the resource to be given. This type of interaction is observed within animal behaviour and is sometimes referred to as proactive prosociality. High learning and low discount rates of the reinforcement learning model are shown to be optimal in order to achieve the outcome that maximises both agents’ reward, and proximity to the given threshold leads to suboptimal learning.
Overview
CoChat Summary
TL;DR
Using multi-agent Q-learning to replay the Sir Philip Sidney game, the authors find that agents usually do not learn Maynard Smith’s honest-signalling equilibrium; instead the Beneficiary typically never signals and the Donor either keeps or gives the resource depending on parameter thresholds. High learning and low discount rates maximize learning of the inclusive-fitness-optimal outcome, while proximity to the threshold reduces it.
Key findings
- In the majority of cases, agents do not learn Maynard Smith’s honest-signalling strategies. The Beneficiary never has an incentive to signal; only the Donor’s behaviour changes with the threshold in Eq 1. Table 4 reports: for r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied), B learns Not signal in 83.3% of runs and D learns Keep in 97.1%; for r=0.8, V=0.1, U=0.9 (satisfied), B learns Not signal in 46.1% and D learns Give in 94.6%; for r=0.9, V=0.75, U=0.95 (satisfied), B learns Not signal in 82.4% and D learns Give in 48.2%.
- The optimal reinforcement-learning parameters are a learning rate of 0.9 and discount rate of 0.1, which produce the expected strategy most often. For Player D in Case 1, the optimal strategy (always keep) is learned in at least 80.9% of iterations; the lowest value, 80.9%, occurs with learning rate 0.1 and discount rate 0.9. For each learning rate, a lower discount rate leads to the optimal strategy being learned a higher proportion of the time.
- Proximity to the threshold produces suboptimal learning. D learns Always Give in Case 2 almost twice as often as in Case 3; the average Q-value difference between Give and Keep is 0.410 for Case 2 and 0.139 for Case 3. In Case 3, D learns Always Give in 48.2% of iterations, and the next most learned strategy is alternating Give/Keep in 38.5%.
- When Eq 1 is satisfied, the emergent {Not signal, Give} outcome is assimilated to proactive prosociality; when it is not satisfied, the outcome is {Not signal, Keep}. Strong signalling is rarely learned by the Beneficiary.
Methods
- The authors model the Sir Philip Sidney game with two agents: a Beneficiary (B) who may be thirsty with probability p or not thirsty, and a Donor (D) who holds an indivisible resource. B’s actions in the simulation are Not signal, Weak signal (c=0.25), and Strong signal (c=0.75); D’s actions are Give and Keep. Payoffs use inclusive fitness, F_i = P_i + rP_j, and the threshold condition is Eq 1: r > (1−U)/(1−V+pV).
- Agents learn via Q-learning. Each dyad plays 500 rounds; the first 50 rounds are exploratory (ε=1), after which exploration decays as ε=2/N. Q-value tables are initialized at 0, and each parameter set is run for 1000 iterations with averages calculated. Learning rates tested are 0.1, 0.5, and 0.9; discount rates are 0.1 and 0.9. Results are checked for p=0 and p=1.
- The analysis compares learned strategies against theoretically derived evolutionarily stable strategies, reports the proportion of runs in which each strategy is learned, and inspects Q-values over rounds. Three main parameter cases are used: r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied); r=0.8, V=0.1, U=0.9 (satisfied); and r=0.9, V=0.75, U=0.95 (satisfied, close to threshold).
Limitations
- Stated: The study focuses on dyadic interactions, leaving open questions about population dynamics, partner selection, and learning over generations with agents initialized from a “parent” agent.
- Stated: Learning and discount rates require interpretation when applied to animal behaviour, whereas the relatedness coefficient and signal costs can be directly understood. The length of the exploration period may also affect results; longer initial exploration or slower ε decay could allow greater separation between Q-values and increase optimal-strategy learning (S5 Appendix).
- Evident from the text: Results are simulation-only, with no empirical animal data; the tested parameter sets and learning/discount rates are limited, and the strategy space is constrained by agent architecture, primarily memory length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
4 Valid · 2 Not valid · 29 Not checked · 95 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Some have addressed this via the replicator-mutator equation [31][32][33], although this still involves switching from one strategy to another within a predefined set, so the strategy space is constrained and chosen from the outset.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
What the sources say
- Supportedcites [31] L Chen, 2024, On the number of equilibria of the replicator-mutator dynamics for noisy social dilemmas, Chaos, Solitons & Fractals. source
Section 2.1 (p. 3) of our copy defines the replicator-mutator framework on 'an infinitely large population consisting of n different strategies S1,...,Sn' whose frequencies sum to 1, and states that 'by means of mutation, individuals in the population might change their strategy to another randomly selected strategy, given by the so-called mutation matrix Q = (q_ji), j,i in {1,...,n}' — i.e. switching only among a pre-specified finite set. The same section (p. 3) confines the dynamics to [0,1]^n on the (n-1)-dimensional simplex with normalisation sum x_i = 1, confirming the strategy space is fixed and chosen from the outset. The citing sentence is a paraphrase of exactly this structure. (LLM, conf=0.90)
- Supportedcites [32] MH Duong, 2019, On equilibrium properties of the replicator–mutator equation in deterministic and random games, Dyn Games Appl. source
The Introduction (p. 1) sets up the replicator–mutator dynamics for a population with n types/strategies S_1,...,S_n fixed at the outset, and defines mutation as individuals 'spontaneously chang[ing] from one strategy to another' via a mutation matrix Q = (q_ji) with j, i ∈ {1,...,n}, where q_ji is the probability that a player of type S_j changes to S_i. Section 2.3 and Eq. (3) reinforce this: switching occurs only among the pre-specified strategies (and, for two-strategy games, only between S_1 and S_2). This matches the claim's paraphrase that mutation is switching within a predefined, constrained strategy space; the evaluative framing ('constrained', 'chosen from the outset') is the citing authors' gloss on a setup the paper states explicitly. (LLM, conf=0.82)
- Not checked yetcites [33] NL Komarova, 2004, Replicator-mutator equation, universality property and population dynamics of learning, J Theor Biol. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the majority of cases, the optimal outcome is one that does not require a signal for the resource to be given.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“proximity to the given threshold leads to suboptimal learning.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We use Multi-Agent Reinforcement Learning (MARL) to show that in the majority of cases, the resulting behaviour adopted by agents is not that shown in the original derivation of the model.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High learning and low discount rates result in the agents learning the optimal strategies in the highest proportion of runs.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 130 claims and to verify the 45 references still open.
- Probably not a substantial claim. 45
- It may state two separate claims as one. 60
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
“In all cases, the learning agents arrive at the optimal strategy a majority of the time;”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F
“| 0.8 | 0.1 | 0.9 | Yes | {N, G} | N = 46.1% | G = 94.6% | | 0.9 | 0.75 | 0.95 | Yes | {N, G} | N = 82.4% | G = 48.2% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Optimal RL parameters
“Figs 3 and 4 show the resulting strategies learned by the RL agents under the parameters S = 0.2, V = 0.2, r = 0.5 (Case 1)”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F
“Not thirsty state, r = 0.5, U = 0.2, V = 0.2.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Revised equilibria of the Philip Sidney game
“provided that c ≤ 0 and V ≤ 0, which must be satisfied: for instance, for p = 1 (Table 1), it must always be true that 1+rU > 1-c+rU.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
“where 0 < V < 1.”
Citations
Cites
Mate selection—A selection for a handicap
Zahavi, A. (1975) · Journal of Theoretical Biology · doi:10.1016/0022-5193(75)90111-3 · Cited by 5,251 · cited 4×
“The study of signalling within dyadic interactions, both involving conflicting or common interest, has given rise to a number of theories surrounding the emergence of honest signalling [3,4].”
InfluentialBiological signals as handicaps
Grafen, A. (1990) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(05)80088-8 · Cited by 2,462 · cited 2×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Reciprocal food sharing in the vampire bat
Wilkinson, G. S. (1984) · Nature · doi:10.1038/308181a0 · Cited by 1,016 · cited 1×
“Much of the literature has come from the field of animal behaviour, where signals are crucial to interactions such as parent-offspring begging [5], food sharing [6,7] or sexual selection [3,8].”
Zahavi, A. (1977) · Journal of Theoretical Biology · doi:10.1016/0022-5193(77)90061-3 · Cited by 1,000 · cited 2×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Smith, J. M., & Harper, D. (2003) · doi:10.1093/oso/9780198526841.001.0001 · Cited by 945
Tambe, M. (1997) · Journal of Artificial Intelligence Research · doi:10.1613/jair.433 · Cited by 897 · cited 9×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
InfluentialLearning Through Reinforcement and Replicator Dynamics
Börgers, T., & Sarin, R. (1997) · Journal of Economic Theory · doi:10.1006/jeth.1997.2319 · Cited by 737 · cited 1×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
The evolutionary origin of human hyper-cooperation
Burkart, J. M., Allon, O., Amici, F., Fichtel, C., Finkenwirth, C., Heschl, A., Huber, J., Isler, K., Kosonen, Z. K., Martins, E., et al. (2014) · Nature Communications · doi:10.1038/ncomms5747 · Cited by 396 · cited 2×
“We observe this phenomenon both in the animal realm [53,54], and within human behaviour [55], and this behaviour may be caused by a number of mechanisms.”
Enquist, M. (1985) · Animal Behaviour · doi:10.1016/s0003-3472(85)80175-5 · Cited by 328 · cited 1×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Watson, R. A., & Szathmáry, E. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.11.009 · Cited by 324 · cited 1×
“Watson et al. [26] also discuss the equivalence of evolutionary processes and simple learning processes, highlighting the use of past experience to make decisions in both types of systems.”
Research works printed but not linked: 5
- Lewis DK. Convention: A Philosophical Study. Cambridge: Harvard University Press; 1969.
- Smit J, Santos FP. Fairness and cooperation between independent reinforcement learners through indirect reciprocity. In: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. AAMAS’24; 2024. p. 2468–70.
- Anastassacos N, García J, Hailes S, Musolesi M. Cooperation and reputation dynamics with reinforcement learning. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. AAMAS’21; 2021. p. 115–23.
- Tuyls K, Lenaerts T, Verbeeck K, Maes S, Manderick B. Towards a relation between learning agents and evolutionary dynamics. In: Proceedings of the Fourteenth Belgium-Netherlands Conference on Artificial Intelligence; 2002. p. 315–22.
- CJCH Watkins, 1992, Q-learning, Machine Learning., 8, 279
Cited by
Li, Z., Liu, R., & Tu, C. (2026) · Ecological Modelling · doi:10.1016/j.ecolmodel.2026.111650 · Cited by 4
“The next generation of computational inquiry is poised to transcend fixed-rule heuristics through the integration of Deep Reinforcement Learning (DRL) into agent-based architectures 134 .”
Metadata
- Authors
- Macmillan-Scott, O., & Musolesi, M.
- Affiliations
- University College London (GB)
- Corresponding author
- Macmillan-Scott, Olivia
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2025-08-26
- Volume
- 21
- Issue
- 8
- Pages
- e1013302
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC12440204
- OpenAlex
- W4413636935
- Cited by
- 1
- References
- 49
- Citations by year
- 2026: 1
- FWCI
- 1.73
- Type
- journal article
- Language
- en
- Update notices
- new_version (2025-09-16)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Evolutionary Psychology and Human Behavior
- Keywords
- honest signaling, multi-agent reinforcement learning, animal communication, evolutionary dynamics, emergent behavior, learning rate, discount rate, dyadic interaction
- MeSH terms
- Animals, Animal Communication, Behavior, Animal, Biological Evolution, Game Theory, Humans, Learning, Models, Biological, Reinforcement, Psychology, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour. https://app.cochat.ai/papers/10.1371/journal.pcbi.1013302
- The paper
- Macmillan-Scott, O., & Musolesi, M. (2025). Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1013302
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.