POET: open-ended coevolution of environments and their optimized solutions
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
How can progress in machine learning and reinforcement learning be automated to generate its own never-ending curriculum of challenges without human intervention? The recent emergence of quality diversity (QD) algorithms offers a glimpse of the potential for such continual open-ended invention. For example, novelty search showcases the benefits of explicit novelty pressure, MAP-Elites and Innovation Engines highlight the advantage of explicit elitism within niches in an otherwise divergent process, and minimal criterion coevolution (MCC) reveals that problems and solutions can coevolve divergently. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper combines these principles to produce a practical approach to generating an endless progression of diverse and increasingly challenging environments while at the same time explicitly optimizing their solutions. An intriguing implication is the opportunity to transfer solutions among environments, reflecting the view that innovation is a circuitous and unpredictable process. POET is tested in a 2-D obstacles course domain, where it generates diverse and sophisticated behaviors that create and solve a wide range of environmental challenges, many of which cannot be solved by direct optimization, or by a direct-path curriculum-building control algorithm. We hope that POET will inspire a new push towards open-ended discovery across many domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The central novelty in this work is the application of agentenvironment co-adaptation 26,27 to generate an agent capable of robust real-time cultural transmission.”
From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [26]
In the citing paper
“Importantly, performance at in-context learning improves with model scale [15,124], implying that methods relying upon this capability will benefit from continuing progress in LLM training.”
From Language Model Crossover: Variation through Few-Shot Prompting (2024) · cites this paper as [124]
In the citing paper
“Supporting this idea, when trained over a large amount of diverse examples, LLMs demonstrate an increasing capability for in-context learning (i.e.inferring novel associations within the input given at test-time when generating completions) [13,15,124].”
From Language Model Crossover: Variation through Few-Shot Prompting (2024) · cites this paper as [124]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3×
“An approach called the Innovation Engine [43] helps to cement the power of goal switching: It can evolve a wide range of images in a single run that are recognized with high-confidence as different image classes by a high-performing deep neural network trained on ImageNet [31].”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“An approach called the Innovation Engine [43] helps to cement the power of goal switching: It can evolve a wide range of images in a single run that are recognized with high-confidence as different image classes by a high-performing deep neural network trained on ImageNet [31].”
BackgroundHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 3×
“Policy gradient methods [50], Q-learning [41], genetic algorithms, other variants of ES, diversity-promoting algorithms such as QD [9, 37, 42, 45] and NS [38], including NS-/NSRA-ES [8], and many other such algorithms are all viable alternatives.”
BackgroundMethodReinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 1×
“In reinforcement learning (RL) [59], modest beginnings in pole balancing [1] have given way to learning to play Atari from pixels [3].”
BackgroundRobots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979 · cited 6×
“Novelty search with local competition (NSLC) [37] and MAP-Elites [9, 42] rectify this limitation by adding a quality pressure (hence ushering in QD in earnest), but they still inevitably hit the limit of what is possible to discover in the particular environment where they are run–there are only so many novel ways a h…”
BackgroundMethodInfluentialSelf-citationAbandoning Objectives: Evolution Through the Search for Novelty Alone
Lehman, J., & Stanley, K. O. (2011) · Evolutionary Computation · doi:10.1162/evco_a_00025 · Cited by 917
Self-citationCompositional pattern producing networks: A novel abstraction of development
Stanley, K. O. (2007) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-007-9028-8 · Cited by 713 · cited 1×
“Much more flexible, or even unbounded environment encodings, such as an indirect encoding like compositional patternproducing networks (CPPNs) [56], can potentially enable POET to traverse a far richer problem space.”
BackgroundSelf-citationSearch-Based Procedural Content Generation: A Taxonomy and Survey
Togelius, J., Yannakakis, G. N., Stanley, K. O., & Browne, C. (2011) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2011.2148116 · Cited by 706 · cited 1×
“The field of procedural content generation (PCG) [52, 61] also offers inspiration for generating new challenges (usually focused on gaming).”
BackgroundSelf-citationEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · cited 1×
“[23] in environments created by humans, but POET could invent its own creative parkour challenges and their solutions.”
BackgroundLearning to control an inverted pendulum using neural networks
Anderson, C. W. (1989) · IEEE Control Systems Magazine · doi:10.1109/37.24809 · Cited by 540 · cited 1×
“In reinforcement learning (RL) [59], modest beginnings in pole balancing [1] have given way to learning to play Atari from pixels [3].”
BackgroundProcedural Content Generation in Games
Shaker, N., Togelius, J., & Nelson, M. J. (2016) · Computational Synthesis and Creative Systems · doi:10.1007/978-3-319-42716-4 · Cited by 519
Diversity is All You Need: Learning Skills without a Reward Function
Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1802.06070 · Cited by 480 · cited 1×
“[13], implementing an NS-archive-like mechanism to overcome reward sparsity by Savinov et al.”
MethodIncremental Evolution of Complex General Behavior
Gomez, F., & Miikkulainen, R. (1997) · Adaptive Behavior · doi:10.1177/105971239700500305 · Cited by 428
Evolving a diversity of virtual creatures through novelty search and local competition
Lehman, J., & Stanley, K. O. (2011) · Proceedings of the 13th annual conference on Genetic and evolutionary computation · doi:10.1145/2001576.2001606 · Cited by 385 · cited 6×
“CCS CONCEPTS • Computing methodologies→ Artificial life; Evolutionary robotics; Reinforcement learning; Neural networks; KEYWORDS Open-ended evolution, coevolution, evolution strategies, novelty search, artificial life ACM Reference Format: Rui Wang, Joel Lehman, Jeff Clune*, and Kenneth O. Stanley*. 2019.”
BackgroundMethodInfluentialSelf-citationGlasmachers, T., Wierstra, D., Schaul, T., Schmidhuber, J., & Yi, S. (2008) · Cited by 355
Parameter-exploring policy gradients
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., & Schmidhuber, J. (2010) · Neural Networks · doi:10.1016/j.neunet.2009.12.004 · Cited by 236 · cited 2×
“Exploring Policy Gradients (PEPG) [51].”
Rechenberg, I. (1978) · Medizinische Informatik und Statistik · doi:10.1007/978-3-642-81283-5_8 · Cited by 223
Cheney, N., MacCurdy, R., Clune, J., & Lipson, H. (2013) · Proceedings of the 15th annual conference on Genetic and evolutionary computation · doi:10.1145/2463372.2463404 · Cited by 211
Self-citationGo-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205 · cited 2×
“of storing diverse state-space discoveries in a QD-like manner in Go-Explore [12] so that the search can return to them later.”
BackgroundSelf-citationRevising the evolutionary computation abstraction: minimal criteria novelty search
Lehman, J., & Stanley, K. O. (2010) · Proceedings of the 12th annual conference on Genetic and evolutionary computation · doi:10.1145/1830483.1830503 · Cited by 177
Self-citationWhy Greatness Cannot Be Planned: The Myth of the Objective
Stanley, K. O., & Lehman, J. (2015) · doi:10.1007/978-3-319-15524-1 · Cited by 139
Self-citationIdeal Evaluation from Coevolution
Jong, E. D. D., & Pollack, J. B. (2004) · Evolutionary Computation · doi:10.1162/106365604773955139 · Cited by 137 · cited 2×
“From an evolutionary perspective, one way to think about learning environments evolving along with their solutions is through the coevolution [10, 14, 26, 44, 62] of the two.”
BackgroundFicici, S. G., & Pollack, J. B. (1998) · Cited by 131
Open-Ended Evolution: Perspectives from the OEE Workshop in York
Taylor, T., Bedau, M., Channon, A., Ackley, D., Banzhaf, W., Beslon, G., Dolson, E., Froese, T., Hickinbotham, S., Ikegami, T., et al. (2016) · Artificial Life · doi:10.1162/artl_a_00210 · Cited by 111 · cited 1×
“such a process could continue indefinitely and with increasing complexity, QD begins to look like a conduit to open-ended evolution [2, 32, 55, 58, 60].”
BackgroundSelf-citationConti, E., Madhavan, V., Such, F. P., Lehman, J., Stanley, K. O., & Clune, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06560 · Cited by 108
Self-citationSoros, L. B., & Stanley, K. O. (2014) · IEEE Symposium on Artificial Life · doi:10.7551/978-0-262-32621-6-ch128 · Cited by 102 · cited 1×
“Unlike conventional coevolutionary algorithms that are usually divided between competitive coevolution [14] and cooperative coevolution [62], MCC introduces a new kind of coevolution that evolves two interlocking populations whose members earn the right to reproduce by satisfying aminimal criterion [35, 53, 54] with r…”
BackgroundSelf-citationOPEN-ENDED ARTIFICIAL EVOLUTION
STANDISH, R. K. (2003) · International Journal of Computational Intelligence and Applications · doi:10.1142/s1469026803000914 · Cited by 91 · cited 1×
“such a process could continue indefinitely and with increasing complexity, QD begins to look like a conduit to open-ended evolution [2, 32, 55, 58, 60].”
BackgroundThe Incremental Pareto-Coevolution Archive
de Jong, E. D. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24854-5_55 · Cited by 80
On the deleterious effects of a priori objectives on evolution and representation
Woolley, B. G., & Stanley, K. O. (2011) · Proceedings of the 13th annual conference on Genetic and evolutionary computation · doi:10.1145/2001576.2001707 · Cited by 63 · cited 1×
“Beginning with the introduction of novelty search [38], we have seen in the field a steady loss of confidence in the idea that we can reliably set objectives, or even know what the right objectives are, and pursue them to ambitious ends [65].”
BackgroundSelf-citationES is more than just a traditional finite-difference approximator
Lehman, J., Chen, J., Clune, J., & Stanley, K. O. (2018) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3205455.3205474 · Cited by 63
Self-citationMinimal criterion coevolution: a new approach to open-ended search
Brant, J. C., & Stanley, K. O. (2017) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3071178.3071186 · Cited by 47
Self-citationNguyen, A., Yosinski, J., & Clune, J. (2016) · Evolutionary Computation · doi:10.1162/evco_a_00189 · Cited by 45 · cited 9×
“optimizing on them only from scratch; (2) neither can they be found through a curriculum-based process aimed at gradually building up to the same challenges POET invented and solved; (3) periodic transfer attempts of solutions from some environments to others–also known as “goal switching” [43]–is important for POET’s…”
BackgroundInfluentialSelf-citationCurriculum Learning for Motor Skills
Karpathy, A., & van de Panne, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30353-1_31 · Cited by 42
The arcade learning environment
BellemareMarc, G., NaddafYavar, VenessJoel, & BowlingMichael (2013) · doi:10.5555/2566972.2566979 · Cited by 40
Does Aligning Phenotypic and Genotypic Modularity Improve the Evolution of Neural Networks?
Huizinga, J., Mouret, J.-B., & Clune, J. (2016) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2908812.2908836 · Cited by 24 · cited 6×
“POET can also be extended to take a more explicit approach to optimizing for generality, such as in CMOEA [24].”
BackgroundInfluentialSelf-citationHuizinga, J., & Clune, J. (2022) · Evolutionary Computation · doi:10.1162/evco_a_00301 · Cited by 20
Self-citation
Research works printed but not linked: 25
- Bedau Mark, 2008, Proceedings of the Eleventh International Conference on Artificial Life (Alife XI), Seth Bullock, Jason Noble, Richard Watson, and Mark Bedau (Eds.). MIT Press
- Brockman Greg, 2016, arXiv:arXiv:1606.01540 · arXiv
- Florensa Carlos, 2018, International Conference on Machine Learning. 1514--1523
- Florensa Carlos, 2017, Proceedings of the 1st Annual Conference on Robot Learning. 482--495
- Forestier Sébastien, 2017, Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint arXiv:1708.02190 · arXiv
- Gupta Abhishek, 2018, Unsupervised Meta-Learning for Reinforcement Learning. arXiv
- David Ha., 2018, Reinforcement Learning for Improving Agent Design. arXiv preprint arXiv:1810.03779 · arXiv
- Heess Nicolas, 2017, Emergence of Locomotion Behaviours in Rich Environments. CoRR abs/1707.02286 · arXiv
- Justesen Niels, 2018, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi.
- Kingma Diederik, 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 · arXiv
- Langdon W. B., 2005, Pfeiffer - A Distributed Open-ended Evolutionary System. In AISB'05: Proceedings of the Joint Symposium on Socially Inspired Computing (METAS 2005
- Lehman Joel, 2011, Stanley
- Matiisen Tambet, 2017, Teacher-Student Curriculum Learning. arXiv preprint arXiv:1707.00183 · arXiv
- Mnih Volodymyr, 2016, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
- Mouret Jean-Baptiste, 2015, Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 · arXiv
- Popovici Elena, 2012, De Jong
- Pugh Justin K, 2016, Stanley
- Salimans Tim, 2017, Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 · arXiv
- Savinov Nikolay, 2018, EPISODIC CURIOSITY THROUGH REACHABILITY. arXiv preprint arXiv:1810.0227
- Schmidhuber Jürgen, 2013, Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology 4
- Schulman John, 2015, International Conference on Machine Learning. 1889--1897
- Soros L.B., 2016, How the Strictness of the Minimal Criterion Impacts Open-Ended Evolution. In ALIFE 15: The Fifteenth International Conference on the Synthesis and Simulation of Living Systems. 208--215
- Stanley Kenneth O., 2017, Open-endedness: The last grand challenge you've never heard of. O'Reilly Online
- Wiegand R. Paul, Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation (GECCO'01)
- Zhang Xingwen, 2017, Stanley
Cited by
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
“This is a very interesting idea encompassing papers such as POET [11], Generative Teaching Networks [12], and the Synthetic Petri Dish [13] which describe algorithms that learn the environment to learn from.”
BackgroundMethodDeep problems with neural network models of human vision
Bowers, J. S., Malhotra, G., Dujmović, M., Llera Montero, M., Tsvetkov, C., Biscione, V., Puebla, G., Adolfi, F., Hummel, J. E., Heaton, R. F., et al. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002813 · Cited by 204
A biological perspective on evolutionary computation
Miikkulainen, R. P., & Forrest, S. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00278-8 · Cited by 129
“In such a setting, collaborative co-evolution can create cooperative structures that work together 23,69 , and competitive co-evolution can dynamically produce new challenges at different levels 102,114 .”
BackgroundEvolutionary Reinforcement Learning: A Survey
Bai, H., Cheng, R., & Jin, Y. (2023) · Intelligent Computing · doi:10.34133/icomputing.0025 · Cited by 137
“The paired open-ended trailblazer (POET) algorithm has been proposed to generate increasingly complex environments and optimize their solutions concurrently by combining the methods of NS, MAP-Elites, and minimal criterion coevolution [170, 171].”
MethodLanguage Model Crossover: Variation through Few-Shot Prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2024) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3694791 · Cited by 75
“multi-objective EAs [15, 18], evolutionary strategies [1, 3], or in support of open-ended evolution [85].”
Self-citation
Metadata
- Authors
- Wang, R., Lehman, J., Clune, J., & Stanley, K. O.
- Affiliations
- Uber AI (United States) (US)
- Venue
- Proceedings of the Genetic and Evolutionary Computation Conference
- Publisher
- ACM
- Published
- 2019-07-13
- Pages
- 142-151
- OpenAlex
- W2954700257
- Semantic Scholar
- ac6449fc71c8768e654a5cdd5f3b3fa230fbb523
- Cited by
- 61
- References
- 35
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 12 · 2023: 8 · 2022: 11 · 2021: 9 · 2020: 10 · 2019: 3
- FWCI
- 3.16
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Genetic and Evolutionary Computation Conference (GECCO) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Evolution and Genetic Dynamics
- Keywords
- open-endedness, quality diversity, novelty search, MAP-Elites, environment generation, solution transfer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.