Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-agent Environments
Open accessAbstract
Neuroevolution (NE) has recently proven a competitive alternative to learning by gradient descent in reinforcement learning tasks. However, the majority of NE methods and associated simulation environments differ crucially from biological evolution: the environment is reset to initial conditions at the end of each generation, whereas natural environments are continuously modified by their inhabitants; agents reproduce based on their ability to maximize rewards within a population, while biological organisms reproduce and die based on internal physiological variables that depend on their resource consumption; simulation environments are primarily single-agent while the biological world is inherently multi-agent and evolves alongside the population. In this work we present a method for continuously evolving adaptive agents without any environment or population reset. The environment is a large grid world with complex spatiotemporal resource generation, containing many agents that are each controlled by an evolvable recurrent neural network and locally reproduce based on their internal physiology. The entire system is implemented in JAX, allowing very fast simulation on a GPU. We show that NE can operate in an ecologically-valid non-episodic multi-agent setting, finding sustainable collective foraging strategies in the presence of a complex interplay between ecological and evolutionary dynamics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent advents in deep reinforcement learning and neuroevolution have enabled the study of complex behaviors in simulations, such as foraging and tool-use.
“Recent advents in deep reinforcement learning (DRL) and neuroevolution have enabled the study of complex behaviors in simulations, such as foraging and tool-use (Pérolat et al., 2017;Hamon et al., 2023;Baker et al., 2020).”
From Dynamics of niche construction in adaptable populations evolving in diverse environments (2023) · cites this paper as Hamon et al., 2023
The Abstract reports that neuroevolution produced sustainable collective foraging strategies in a simulated multi-agent grid world, and A.2 describes resource-consumption studies enabled by advances in reinforcement learning. The cited paper does not address tool-use, so it supports the foraging example but not the whole sentence as attributed by the grouped citations. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
Fluctuations in the Abundance of a Species considered Mathematically1
VOLTERRA, V. (1926) · Nature · doi:10.1038/118558a0 · Cited by 2,054 · see citing works
Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019
Auger, A., & Stützle, T. (2019) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · Cited by 1,726 · see citing works
The extended evolutionary synthesis: its structure, assumptions and predictions
Laland, K. N., Uller, T., Feldman, M. W., Sterelny, K., Müller, G. B., Moczek, A., Jablonka, E., & Odling-Smee, J. (2015) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2015.1019 · Cited by 1,399 · see citing works
Allostasis: A model of predictive regulation
Sterling, P. (2012) · Physiology & Behavior · doi:10.1016/j.physbeh.2011.06.004 · Cited by 1,073 · see citing works
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Salimans, T., Ho, J., Chen, X., Sidor, S., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1703.03864 · Cited by 1,027 · see citing works
Designing neural networks through neuroevolution
Stanley, K. O., Clune, J., Lehman, J., & Miikkulainen, R. P. (2018) · Nature Machine Intelligence · doi:10.1038/s42256-018-0006-z · Cited by 702 · see citing works
A Survey on Evolutionary Neural Architecture Search
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Tan, K. C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3100554 · Cited by 693 · see citing works
Such, F. P., Madhavan, V., Conti, E., Lehman, J., Stanley, K. O., & Clune, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06567 · Cited by 689 · see citing works
Post, D. M., & Palkovacs, E. P. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2009.0012 · Cited by 646 · see citing works
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445 · see citing works
On the Measure of Intelligence
Chollet, F. (2019) · arXiv · doi:10.48550/arxiv.1911.01547 · Cited by 218 · see citing works
Multi-agent Reinforcement Learning in Sequential Social DilemmasPreprint
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017) · arXiv (Cornell University) · doi:10.5555/3091125.3091194 · Cited by 151 · see citing works
Gupta, A., Yu, J., Zhao, T. Z., Kumar, V., Rovinsky, A., Xu, K., Devlin, T. J., & Levine, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561384 · Cited by 75 · see citing works
Evolving plastic neural networks with novelty search
Risi, S., Hughes, C. E., & Stanley, K. O. (2010) · Adaptive Behavior · doi:10.1177/1059712310379923 · Cited by 75 · see citing works
Introducing Ecological Dynamics into Common-Pool Resource Experiments
Janssen, M. A. (2010) · Ecology and Society · doi:10.5751/es-03296-150207 · Cited by 71 · see citing works
On the deleterious effects of a priori objectives on evolution and representation
Woolley, B. G., & Stanley, K. O. (2011) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2001576.2001707 · Cited by 64 · see citing works
Evolution and dispersal under climatic instability: a simple evolutionary algorithm
Grove, M. (2014) · Adaptive Behavior · doi:10.1177/1059712314533573 · Cited by 38 · see citing works
Diversity preservation in minimal criterion coevolution through resource limitation
Brant, J. C., & Stanley, K. O. (2020) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3377930.3389809 · Cited by 9 · see citing works
The Hunger Games: Embodied agents evolving foraging strategies on the frugal-greedy spectrum
Aubert-Kato, N., Witkowski, O., & Ikegami, T. (2015) · doi:10.7551/978-0-262-33027-5-ch065 · Cited by 6 · see citing works
Plasticity and evolvability under environmental variability
Nisioti, E., & Moulin-Frier, C. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528826 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 13
- Nicolas Anastassacos, Julian García, Stephen Hailes, and Mirco Musolesi. 2021. Cooperation and Reputation Dynamics with Reinforcement Learning. arXiv:2102.07523 [cs] (Feb. 2021). arXiv:2102.07523 [cs] · arXiv
- Jeff Clune. 2020. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv:1905.10985 [cs] (Jan. 2020). http://arxiv.org/abs/1905.10985 arXiv: 1905.10985. · arXiv
- John D. Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine. 2020. Ecological Reinforcement Learning. arXiv:2006.12478 [cs, stat] (June 2020). http://arxiv.org/abs/2006.12478 arXiv: 2006.12478. · arXiv
- Garrett Hardin. 1968. The Tragedy of the Commons. Science 162, 3859 (1968), 1243-1248.
- Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel. 2019. Autocur- ricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742 (2019). · arXiv
- Gary Marcus and Ernest Davis. 2019. Rebooting AI: building artificial intelligence we can trust (first edition ed.). Pantheon Books, New York.
- Risto Miikkulainen, Eliana Feasley, Leif Johnson, Igor Karpov, Padmini Ra- jagopalan, Aditya Rawal, and Wesley Tansey. 2012. Multiagent Learning through Neuroevolution. In Advances in Computational Intelligence, Jing Liu, Cesare Alippi, Bernadette Bouchon-Meunier, Garrison W. Greenwood, and Hussein A. Abbass (Eds.). Vol. 7311. Springer Berlin Heidelberg, Berlin, Heidelberg, 24-46. https://doi.org/10.1007/978-3-642-30687-7_2 · DOI
- Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. 2018. Time Limits in Reinforcement Learning. In Proceedings of the 35th International Con- ference on Machine Learning. PMLR, 4045-4054. https://proceedings.mlr.press/ v80/pardo18a.html ISSN: 2640-3498.
- G A Parker. 2000. Scramble in Behaviour and Ecology. Philosophical Transactions of the Royal Society B: Biological Sciences 355, 1403 (Nov. 2000), 1637-1645.
- Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. 2017. A multi-agent reinforcement learning model of common- pool resource appropriation. Advances in neural information processing systems 30 (2017).
- Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. 2020. Automatic curriculum learning for deep RL: A short survey. arXiv:2003.04664 [cs, stat] (2020). http://arxiv.org/abs/2003.04664 · arXiv
- Kenneth O. Stanley and Joel Lehman. 2015. Why Greatness Cannot Be Planned: The Myth of the Objective. Springer Publishing Company, Incorporated.
- Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. 2019. Paired Open- Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Di- verse Learning Environments and Their Solutions. arXiv:1901.01753 [cs] (2019). http://arxiv.org/abs/1901.01753 arXiv: 1901.01753. · arXiv
Cited by
Sorted by citations
Dynamics of niche construction in adaptable populations evolving in diverse environments
Nisioti, E., & Moulin-Frier, C. (2023) · The 2023 Conference on Artificial Life · doi:10.1162/isal_a_00668 · Cited by 2 · see citing works
Self-citationLeveraging More of Biology in Evolutionary Reinforcement Learning
Gašperov, B., Ðurasević, M., & Jakobović, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56855-8_6 · Cited by 2 · see citing works
Emergent Kin Selection of Altruistic Feeding via Non-episodic Neuroevolution
Taylor-Davies, M., Hamon, G., Boulet, T., & Moulin-Frier, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90062-4_31
Self-citationA neuroevolution-driven agent-based model of coral larvae settlement
Hofmann, S., Voolstra, C. R., & von Mammen, S. (2026) · PLoS ONE · doi:10.1371/journal.pone.0359316
- Authors
- Hamon, G., Nisioti, E., & Moulin-Frier, C.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- Venue
- Proceedings of the Companion Conference on Genetic and Evolutionary Computation
- Publisher
- ACM
- Published
- 2023-07-15
- Pages
- 143-146
- OpenAlex
- W4385188838
- Cited by
- 4
- References
- 6
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1
- FWCI
- 0.71
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Genetic and Evolutionary Computation Conference Companion (GECCO Companion) (publishedVersion)
- HAL (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Agence National de la Recherche (ANR-20-CE23- 0006)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Evolution and Genetic Dynamics
- Keywords
- neuroevolution, eco-evolutionary dynamics, multi-agent systems, recurrent neural networks, group foraging, resource consumption, grid world, JAX
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
