Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 10 of the 55 sources its claims cite were checked for support.
Abstract
The coevolution of signalling is a complex problem within animal behaviour, and is also central to communication between artificial agents. The Sir Philip Sidney game was designed to model this dyadic interaction from an evolutionary biology perspective, and was formulated to demonstrate the emergence of honest signalling. We use Multi-Agent Reinforcement Learning (MARL) to show that in the majority of cases, the resulting behaviour adopted by agents is not that shown in the original derivation of the model. This paper demonstrates that MARL can be a powerful tool to study evolutionary dynamics and understand the underlying mechanisms of learning over generations; particularly advantageous is the interpretability of this type of approach, as well as the fact that it allows us to study emergent behaviour without the need to constrain the strategy space from the outset. Although it originally set out to exemplify honest signalling, we show that the game provides no incentive for such behaviour. In the majority of cases, the optimal outcome is one that does not require a signal for the resource to be given. This type of interaction is observed within animal behaviour and is sometimes referred to as proactive prosociality. High learning and low discount rates of the reinforcement learning model are shown to be optimal in order to achieve the outcome that maximises both agents’ reward, and proximity to the given threshold leads to suboptimal learning.
Overview
CoChat Summary
TL;DR
Using multi-agent Q-learning to replay the Sir Philip Sidney game, the authors find that agents usually do not learn Maynard Smith’s honest-signalling equilibrium; instead the Beneficiary typically never signals and the Donor either keeps or gives the resource depending on parameter thresholds. High learning and low discount rates maximize learning of the inclusive-fitness-optimal outcome, while proximity to the threshold reduces it.
Key findings
- In the majority of cases, agents do not learn Maynard Smith’s honest-signalling strategies. The Beneficiary never has an incentive to signal; only the Donor’s behaviour changes with the threshold in Eq 1. Table 4 reports: for r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied), B learns Not signal in 83.3% of runs and D learns Keep in 97.1%; for r=0.8, V=0.1, U=0.9 (satisfied), B learns Not signal in 46.1% and D learns Give in 94.6%; for r=0.9, V=0.75, U=0.95 (satisfied), B learns Not signal in 82.4% and D learns Give in 48.2%.
- The optimal reinforcement-learning parameters are a learning rate of 0.9 and discount rate of 0.1, which produce the expected strategy most often. For Player D in Case 1, the optimal strategy (always keep) is learned in at least 80.9% of iterations; the lowest value, 80.9%, occurs with learning rate 0.1 and discount rate 0.9. For each learning rate, a lower discount rate leads to the optimal strategy being learned a higher proportion of the time.
- Proximity to the threshold produces suboptimal learning. D learns Always Give in Case 2 almost twice as often as in Case 3; the average Q-value difference between Give and Keep is 0.410 for Case 2 and 0.139 for Case 3. In Case 3, D learns Always Give in 48.2% of iterations, and the next most learned strategy is alternating Give/Keep in 38.5%.
- When Eq 1 is satisfied, the emergent {Not signal, Give} outcome is assimilated to proactive prosociality; when it is not satisfied, the outcome is {Not signal, Keep}. Strong signalling is rarely learned by the Beneficiary.
Methods
- The authors model the Sir Philip Sidney game with two agents: a Beneficiary (B) who may be thirsty with probability p or not thirsty, and a Donor (D) who holds an indivisible resource. B’s actions in the simulation are Not signal, Weak signal (c=0.25), and Strong signal (c=0.75); D’s actions are Give and Keep. Payoffs use inclusive fitness, F_i = P_i + rP_j, and the threshold condition is Eq 1: r > (1−U)/(1−V+pV).
- Agents learn via Q-learning. Each dyad plays 500 rounds; the first 50 rounds are exploratory (ε=1), after which exploration decays as ε=2/N. Q-value tables are initialized at 0, and each parameter set is run for 1000 iterations with averages calculated. Learning rates tested are 0.1, 0.5, and 0.9; discount rates are 0.1 and 0.9. Results are checked for p=0 and p=1.
- The analysis compares learned strategies against theoretically derived evolutionarily stable strategies, reports the proportion of runs in which each strategy is learned, and inspects Q-values over rounds. Three main parameter cases are used: r=0.5, V=0.2, U=0.2 (Eq 1 not satisfied); r=0.8, V=0.1, U=0.9 (satisfied); and r=0.9, V=0.75, U=0.95 (satisfied, close to threshold).
Limitations
- Stated: The study focuses on dyadic interactions, leaving open questions about population dynamics, partner selection, and learning over generations with agents initialized from a “parent” agent.
- Stated: Learning and discount rates require interpretation when applied to animal behaviour, whereas the relatedness coefficient and signal costs can be directly understood. The length of the exploration period may also affect results; longer initial exploration or slower ε decay could allow greater separation between Q-values and increase optimal-strategy learning (S5 Appendix).
- Evident from the text: Results are simulation-only, with no empirical animal data; the tested parameter sets and learning/discount rates are limited, and the strategy space is constrained by agent architecture, primarily memory length.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
4 Valid · 2 Not valid · 29 Not checked · 95 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Some have addressed this via the replicator-mutator equation [31][32][33], although this still involves switching from one strategy to another within a predefined set, so the strategy space is constrained and chosen from the outset.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
What the sources say
- Supportedcites [31] L Chen, 2024, On the number of equilibria of the replicator-mutator dynamics for noisy social dilemmas, Chaos, Solitons & Fractals. source
Section 2.1 (p. 3) of our copy defines the replicator-mutator framework on 'an infinitely large population consisting of n different strategies S1,...,Sn' whose frequencies sum to 1, and states that 'by means of mutation, individuals in the population might change their strategy to another randomly selected strategy, given by the so-called mutation matrix Q = (q_ji), j,i in {1,...,n}' — i.e. switching only among a pre-specified finite set. The same section (p. 3) confines the dynamics to [0,1]^n on the (n-1)-dimensional simplex with normalisation sum x_i = 1, confirming the strategy space is fixed and chosen from the outset. The citing sentence is a paraphrase of exactly this structure. (LLM, conf=0.90)
- Supportedcites [32] MH Duong, 2019, On equilibrium properties of the replicator–mutator equation in deterministic and random games, Dyn Games Appl. source
The Introduction (p. 1) sets up the replicator–mutator dynamics for a population with n types/strategies S_1,...,S_n fixed at the outset, and defines mutation as individuals 'spontaneously chang[ing] from one strategy to another' via a mutation matrix Q = (q_ji) with j, i ∈ {1,...,n}, where q_ji is the probability that a player of type S_j changes to S_i. Section 2.3 and Eq. (3) reinforce this: switching occurs only among the pre-specified strategies (and, for two-strategy games, only between S_1 and S_2). This matches the claim's paraphrase that mutation is switching within a predefined, constrained strategy space; the evaluative framing ('constrained', 'chosen from the outset') is the citing authors' gloss on a setup the paper states explicitly. (LLM, conf=0.82)
- Not checked yetcites [33] NL Komarova, 2004, Replicator-mutator equation, universality property and population dynamics of learning, J Theor Biol. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the majority of cases, the optimal outcome is one that does not require a signal for the resource to be given.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“proximity to the given threshold leads to suboptimal learning.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We use Multi-Agent Reinforcement Learning (MARL) to show that in the majority of cases, the resulting behaviour adopted by agents is not that shown in the original derivation of the model.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High learning and low discount rates result in the agents learning the optimal strategies in the highest proportion of runs.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 130 claims and to verify the 45 references still open.
- Probably not a substantial claim. 45
- It may state two separate claims as one. 60
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
“In all cases, the learning agents arrive at the optimal strategy a majority of the time;”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F
“| 0.8 | 0.1 | 0.9 | Yes | {N, G} | N = 46.1% | G = 94.6% | | 0.9 | 0.75 | 0.95 | Yes | {N, G} | N = 82.4% | G = 48.2% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Optimal RL parameters
“Figs 3 and 4 show the resulting strategies learned by the RL agents under the parameters S = 0.2, V = 0.2, r = 0.5 (Case 1)”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F U N C O R R E C T E D P R O O F
“Not thirsty state, r = 0.5, U = 0.2, V = 0.2.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > Revised equilibria of the Philip Sidney game
“provided that c ≤ 0 and V ≤ 0, which must be satisfied: for instance, for p = 1 (Table 1), it must always be true that 1+rU > 1-c+rU.”
§ Maynard Smith revisited: A multi-agent reinforcement learning approach to the coevolution of signalling behaviour > U N C O R R E C T E D P R O O F
“where 0 < V < 1.”
Citations
Cites
Mate selection—A selection for a handicap
Zahavi, A. (1975) · Journal of Theoretical Biology · doi:10.1016/0022-5193(75)90111-3 · Cited by 5,251 · cited 4×
“The study of signalling within dyadic interactions, both involving conflicting or common interest, has given rise to a number of theories surrounding the emergence of honest signalling [3,4].”
InfluentialBiological signals as handicaps
Grafen, A. (1990) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(05)80088-8 · Cited by 2,462 · cited 2×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Reciprocal food sharing in the vampire bat
Wilkinson, G. S. (1984) · Nature · doi:10.1038/308181a0 · Cited by 1,016 · cited 1×
“Much of the literature has come from the field of animal behaviour, where signals are crucial to interactions such as parent-offspring begging [5], food sharing [6,7] or sexual selection [3,8].”
Zahavi, A. (1977) · Journal of Theoretical Biology · doi:10.1016/0022-5193(77)90061-3 · Cited by 1,000 · cited 2×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Smith, J. M., & Harper, D. (2003) · doi:10.1093/oso/9780198526841.001.0001 · Cited by 945
Tambe, M. (1997) · Journal of Artificial Intelligence Research · doi:10.1613/jair.433 · Cited by 897 · cited 9×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
InfluentialLearning Through Reinforcement and Replicator Dynamics
Börgers, T., & Sarin, R. (1997) · Journal of Economic Theory · doi:10.1006/jeth.1997.2319 · Cited by 737 · cited 1×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
The evolutionary origin of human hyper-cooperation
Burkart, J. M., Allon, O., Amici, F., Fichtel, C., Finkenwirth, C., Heschl, A., Huber, J., Isler, K., Kosonen, Z. K., Martins, E., et al. (2014) · Nature Communications · doi:10.1038/ncomms5747 · Cited by 396 · cited 2×
“We observe this phenomenon both in the animal realm [53,54], and within human behaviour [55], and this behaviour may be caused by a number of mechanisms.”
Enquist, M. (1985) · Animal Behaviour · doi:10.1016/s0003-3472(85)80175-5 · Cited by 328 · cited 1×
“The Sir Philip Sidney game was developed by Maynard Smith as a way to model signalling within animal behaviour [12] - in particular, the aim of the model was to illustrate the mechanisms of honest signalling initially proposed by Zahavi [3,9] and subsequently developed by others [10,47].”
Watson, R. A., & Szathmáry, E. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2015.11.009 · Cited by 324 · cited 1×
“Watson et al. [26] also discuss the equivalence of evolutionary processes and simple learning processes, highlighting the use of past experience to make decisions in both types of systems.”
Honest signalling: the Philip Sidney game
Smith, J. M. (1991) · Animal Behaviour · doi:10.1016/s0003-3472(05)80161-7 · Cited by 311
Sexual selection: the handicap principle does work – sometimes
Pomiankowski, A. (1987) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1987.0038 · Cited by 287
Efficiency in evolutionary games: Darwin, nash and the secret handshake
Robson, A. J. (1990) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(05)80082-7 · Cited by 281 · cited 2×
“The study of signalling within dyadic interactions, both involving conflicting or common interest, has given rise to a number of theories surrounding the emergence of honest signalling [3,4].”
The future of theoretical evolutionary game theory
Traulsen, A., & Glynatsi, N. E. (2023) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0508 · Cited by 173 · cited 1×
“The value of interactions between the disciplines of evolutionary game theory and machine learning to gain new insights about evolutionary dynamics of interacting agents has previously been discussed [27], in particular with regards to multi-agent systems [28].”
Evolutionary game theory and multi-agent reinforcement learning
TUYLS, K., & NOWÉ, A. (2005) · The Knowledge Engineering Review · doi:10.1017/s026988890500041x · Cited by 128 · cited 1×
“The value of interactions between the disciplines of evolutionary game theory and machine learning to gain new insights about evolutionary dynamics of interacting agents has previously been discussed [27], in particular with regards to multi-agent systems [28].”
A selection-mutation model for q-learning in multi-agent systems
Tuyls, K., Verbeeck, K., & Lenaerts, T. (2003) · Proceedings of the second international joint conference on Autonomous agents and multiagent systems · doi:10.1145/860575.860687 · Cited by 120
The continuous Sir Philip Sidney game: a simple model of biological signalling
Johnstone, R. A., & Grafen, A. (1992) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(05)80674-5 · Cited by 118 · cited 6×
“…designed to model animal behaviour, and therefore simulates learning over generations, we consider inclusive fitness rather than individual payoff; the use of inclusive fitness instead of individual reward is in line with Maynard Smith’s original formulation [12] and subsequent studies [6,30,34].”
InfluentialThe evolution of begging: signaling and sibling competition
Rodríguez-Gironés, M. A., Cotton, P. A., & Kacelnik, A. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.25.14637 · Cited by 95 · cited 2×
“One example of such a strategy combination is found in Rodr´ıguez-Giron´es et al.’s [52] model of begging between offspring and parents: they find an equilibrium where chicks do not beg for food, and parents still provide optimally.”
The power of associative learning and the ontogeny of optimal behaviour
Enquist, M., Lind, J., & Ghirlanda, S. (2016) · Royal Society Open Science · doi:10.1098/rsos.160734 · Cited by 94
Some dynamics of signaling games
Huttegger, S. M., Skyrms, B., Tarrès, P., & Wagner, E. O. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1400838111 · Cited by 92 · cited 2×
“Signalling is an important mechanism of information transfer between organisms [1], and is also a central aspect of communication [2].”
Signalling among relatives. I. Is costly signalling too costly?
Bergstrom, C. T., & Lachmann, M. (1997) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.1997.0041 · Cited by 89 · cited 7×
“This result has previously been discussed in the literature [48,49,52], and relaxes the assumption of conditionality: the equilibrium shown in this paper is reached when we remove the condition that the resource cannot be given in the absence of a signal.”
Influentiald’Ettorre, P., & Hughes, D. P. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199216840.001.0001 · Cited by 83
Replicator-mutator equation, universality property and population dynamics of learning
Komarova, N. L. (2004) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2004.05.004 · Cited by 82 · cited 1×
“Some have addressed this via the replicator–mutator equation [31–33], although this still involves switching from one strategy to another within a predefined set, so the strategy space is constrained and chosen from the outset.”
Horn, L., Scheer, C., Bugnyar, T., & Massen, J. J. M. (2016) · Biology Letters · doi:10.1098/rsbl.2016.0649 · Cited by 76 · cited 2×
“We observe this phenomenon both in the animal realm [53,54], and within human behaviour [55], and this behaviour may be caused by a number of mechanisms.”
The complexity of human cooperation under indirect reciprocity
Santos, F. P., Pacheco, J. M., & Santos, F. C. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2020.0291 · Cited by 76 · cited 1×
“Focusing on prosocial human behaviour, others have identified mechanisms like reputation to be central to indirect reciprocity and to cooperation more widely [15].”
Dynamic stability and basins of attraction in the Sir Philip Sidney game
Huttegger, S. M., & Zollman, K. (2010) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2009.2105 · Cited by 69 · cited 7×
“Hamblin and Hurd [49] refer to this equilibrium in their work as the ‘non-communicating, always-give’ set, reached by Huttegger and Zollman’s [30] aforementioned ‘donate only if no signal’ strategy.”
InfluentialEnriching behavioral ecology with reinforcement learning methods
Frankenhuis, W. E., Panchanathan, K., & Barto, A. G. (2018) · Behavioural Processes · doi:10.1016/j.beproc.2018.01.008 · Cited by 68 · cited 2×
“Frankenhuis, Panchanathan and Barto [25] highlight the valuable insights that can be drawn from applying reinforcement learning methods to behavioural ecology, both across generations and within organisms, for instance to study adaptive behaviour based on experience.”
Emergent behaviours in multi-agent systems with Evolutionary Game Theory
Han, T. A. (2022) · AI Communications · doi:10.3233/aic-220104 · Cited by 65 · cited 1×
“Evolutionary game theory in conjunction with computational approaches has proven fruitful in studying emergent collective behaviour more widely, although there is still uncertainty relating to the mechanisms leading to behaviour akin to altruism [24].”
Identification of animal behavioral strategies by inverse reinforcement learning
Yamaguchi, S., Naoki, H., Ikeda, M., Tsukada, Y., Nakano, S., Mori, I., & Ishii, S. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006122 · Cited by 61 · cited 1×
“Reinforcement learning has been discussed as a useful tool to study animal behaviour and to model their learning through trial and error [25,40–42].”
Learning to flock through reinforcement
Durve, M., Peruani, F., & Celani, A. (2020) · Physical Review E · doi:10.1103/physreve.102.012601 · Cited by 58
Marmoset prosociality is intentional
Burkart, J. M., & van Schaik, C. P. (2020) · Animal Cognition · doi:10.1007/s10071-020-01363-6 · Cited by 54 · cited 3×
“We observe this phenomenon both in the animal realm [53,54], and within human behaviour [55], and this behaviour may be caused by a number of mechanisms.”
Begging and parent–offspring conflict in grey seals
Smiseth, P. T., & Lorentsen, S. (2001) · Animal Behaviour · doi:10.1006/anbe.2001.1763 · Cited by 41 · cited 1×
“Much of the literature has come from the field of animal behaviour, where signals are crucial to interactions such as parent-offspring begging [5], food sharing [6,7] or sexual selection [3,8].”
The Dynamics of Costly Signaling
Wagner, E. O. (2013) · Games · doi:10.3390/g4020163 · Cited by 33 · cited 1×
“There is considerable work in this area; previous studies have precisely looked at signalling games using replicator dynamics [1,29], in particular at the Sir Philip Sidney game [30].”
Why not lie? Costs enforce honesty in an experimental signalling game
Polnaszek, T. J., & Stephens, D. W. (2013) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2013.2457 · Cited by 32 · cited 1×
“The handicap principle has been shown both theoretically [10], and experimentally [11]: using the example of blue jays, honesty persisted with high signal costs, and disappeared with low costs.”
Bauer, J., Broom, M., & Alonso, E. (2019) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2019.0355 · Cited by 29 · cited 1×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
Communicating with Unknown Teammates
Barrett, S., Agmon, N., Hazon, N., Kraus, S., & Stone, P. (2014) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-419-0-45 · Cited by 25 · cited 1×
“…designed to model animal behaviour, and therefore simulates learning over generations, we consider inclusive fitness rather than individual payoff; the use of inclusive fitness instead of individual reward is in line with Maynard Smith’s original formulation [12] and subsequent studies [6,30,34].”
The limits and robustness of reinforcement learning in Lewis signalling games
Catteeuw, D., & Manderick, B. (2014) · Connection Science · doi:10.1080/09540091.2014.885303 · Cited by 24 · cited 1×
“Catteeuw and Manderick [13] used a reinforcement learning approach to study equilibria in Lewis signalling games [14], where they showed how the use of this method can lead to optimal equilibria.”
Communication, Commitment, and Deception in Social Dilemmas: Experimental EvidencePreprint
Camera, G., Casari, M., & Bigoni, M. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1854132 · Cited by 23 · cited 2×
“When it comes to scenarios where there is pure conflict of interest, the incentive to communicate disappears [22].”
A model for the evolution of reinforcement learning in fluctuating games
Dridi, S., & Lehmann, L. (2015) · Animal Behaviour · doi:10.1016/j.anbehav.2015.01.037 · Cited by 22 · cited 1×
“Reinforcement learning has been discussed as a useful tool to study animal behaviour and to model their learning through trial and error [25,40–42].”
Evolution of honest signaling by social punishment
Catteeuw, D., Han, T. A., & Manderick, B. (2014) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2576768.2598312 · Cited by 19 · cited 2×
“Whereas replicator dynamics is generally concerned with infinite populations, some analyses of the Sir Philip Sidney game on finite populations have also studied the evolutionary stability of honest signalling [34,35].”
Prosocial dynamics in multiagent systems
Santos, F. P. (2024) · AI Magazine · doi:10.1002/aaai.12143 · Cited by 17 · cited 1×
“Open questions remain regarding how to achieve prosociality among artificial agents in social dilemma-type situations [23].”
Emergent Escape-based Flocking behavior using Multi-Agent Reinforcement Learning
Hahn, C., Phan, T., Gabor, T., Belzner, L., & Linnhoff–Popien, C. (2019) · Conference on Artificial Life (ALIFE) · doi:10.1162/isal_a_00226.xml · Cited by 15
When will evolution lead to deceptive signaling in the Sir Philip Sidney game?
Hamblin, S., & Hurd, P. L. (2009) · Theoretical Population Biology · doi:10.1016/j.tpb.2009.02.002 · Cited by 13 · cited 6×
“Hamblin and Hurd [49] refer to this equilibrium in their work as the ‘non-communicating, always-give’ set, reached by Huttegger and Zollman’s [30] aforementioned ‘donate only if no signal’ strategy.”
InfluentialYamada, J., Shawe-Taylor, J., & Fountas, Z. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206765 · Cited by 11 · cited 1×
“In the multi-agent setting, it has been used to study animal behaviour in areas including predatory-prey dynamics [43,44] and collective/swarm movement [45,46].”
Strategic inattention in the Sir Philip Sidney Game
Whitmeyer, M. (2020) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2020.110513 · Cited by 8 · cited 1×
“This framework has been the subject of several studies, particularly within the biological sciences [6,48–50].”
On the number of equilibria of the replicator-mutator dynamics for noisy social dilemmas
Chen, L., Deng, C., Duong, M. H., & Han, T. A. (2024) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2024.114565 · Cited by 7 · cited 1×
“Some have addressed this via the replicator–mutator equation [31–33], although this still involves switching from one strategy to another within a predefined set, so the strategy space is constrained and chosen from the outset.”
On equilibrium properties of the replicator–mutator equation in deterministic and random games
Duong, M. H., & Han, T. A. (2020) · Dynamic Games and Applications · doi:10.1007/s13235-019-00338-8 · Cited by 7
Evolutionary stability of honest signaling in finite populations
Catteeuw, D., Manderick, B., & Han, T. A. (2013) · 2013 IEEE Congress on Evolutionary Computation · doi:10.1109/cec.2013.6557917 · Cited by 5 · cited 3×
“At time t , each player has an action a and state s - this state is made up of both players’ actions in the previous round.”
Emergence of Collaborative Hunting via Multi-Agent Deep Reinforcement Learning
Tsutsui, K., Takeda, K., & Fujii, K. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-37660-3_15 · Cited by 5 · cited 1×
“The value of p does not affect the results - this was demonstrated by Maynard Smith in the original formulation of the model [12], and we find that this holds for reinforcement learning agents.”
Anastassacos, N., Hailes, S., & Musolesi, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6190 · Cited by 4 · cited 1×
“Further extensions to this research could explore population dynamics, perhaps including partner selection (see for example [57]).”
Self-citation
Research works printed but not linked: 5
- Lewis DK. Convention: A Philosophical Study. Cambridge: Harvard University Press; 1969.
- Smit J, Santos FP. Fairness and cooperation between independent reinforcement learners through indirect reciprocity. In: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. AAMAS’24; 2024. p. 2468–70.
- Anastassacos N, García J, Hailes S, Musolesi M. Cooperation and reputation dynamics with reinforcement learning. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. AAMAS’21; 2021. p. 115–23.
- Tuyls K, Lenaerts T, Verbeeck K, Maes S, Manderick B. Towards a relation between learning agents and evolutionary dynamics. In: Proceedings of the Fourteenth Belgium-Netherlands Conference on Artificial Intelligence; 2002. p. 315–22.
- CJCH Watkins, 1992, Q-learning, Machine Learning., 8, 279
Cited by
Li, Z., Liu, R., & Tu, C. (2026) · Ecological Modelling · doi:10.1016/j.ecolmodel.2026.111650 · Cited by 5
“The next generation of computational inquiry is poised to transcend fixed-rule heuristics through the integration of Deep Reinforcement Learning (DRL) into agent-based architectures 134 .”
Metadata
- Authors
- Macmillan-Scott, O., & Musolesi, M.
- Affiliations
- University College London (GB)
- Corresponding author
- Macmillan-Scott, Olivia
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2025-08-26
- Volume
- 21
- Issue
- 8
- Pages
- e1013302
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC12440204
- OpenAlex
- W4413636935
- Cited by
- 1
- References
- 49
- Citations by year
- 2026: 1
- FWCI
- 1.73
- Type
- journal article
- Language
- en
- Update notices
- new_version (2025-09-16)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Evolutionary Psychology and Human Behavior
- Keywords
- honest signaling, multi-agent reinforcement learning, animal communication, evolutionary dynamics, emergent behavior, learning rate, discount rate, dyadic interaction
- MeSH terms
- Animals, Animal Communication, Behavior, Animal, Biological Evolution, Game Theory, Humans, Learning, Models, Biological, Reinforcement, Psychology, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.