The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
POET: open-ended coevolution of environments and their optimized solutions
Open accessAbstract
How can progress in machine learning and reinforcement learning be automated to generate its own never-ending curriculum of challenges without human intervention? The recent emergence of quality diversity (QD) algorithms offers a glimpse of the potential for such continual open-ended invention. For example, novelty search showcases the benefits of explicit novelty pressure, MAP-Elites and Innovation Engines highlight the advantage of explicit elitism within niches in an otherwise divergent process, and minimal criterion coevolution (MCC) reveals that problems and solutions can coevolve divergently. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper combines these principles to produce a practical approach to generating an endless progression of diverse and increasingly challenging environments while at the same time explicitly optimizing their solutions. An intriguing implication is the opportunity to transfer solutions among environments, reflecting the view that innovation is a circuitous and unpredictable process. POET is tested in a 2-D obstacles course domain, where it generates diverse and sophisticated behaviors that create and solve a wide range of environmental challenges, many of which cannot be solved by direct optimization, or by a direct-path curriculum-building control algorithm. We hope that POET will inspire a new push towards open-ended discovery across many domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The central novelty in this work is the application of agentenvironment co-adaptation 26,27 to generate an agent capable of robust real-time cultural transmission.”
From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [26]
In the citing paper
“Importantly, performance at in-context learning improves with model scale [15,124], implying that methods relying upon this capability will benefit from continuing progress in LLM training.”
From Language Model Crossover: Variation through Few-Shot Prompting (2024) · cites this paper as [124]
In the citing paper
“Supporting this idea, when trained over a large amount of diverse examples, LLMs demonstrate an increasing capability for in-context learning (i.e.inferring novel associations within the input given at test-time when generating completions) [13,15,124].”
From Language Model Crossover: Variation through Few-Shot Prompting (2024) · cites this paper as [124]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3×
“An approach called the Innovation Engine [43] helps to cement the power of goal switching: It can evolve a wide range of images in a single run that are recognized with high-confidence as different image classes by a high-performing deep neural network trained on ImageNet [31].”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“An approach called the Innovation Engine [43] helps to cement the power of goal switching: It can evolve a wide range of images in a single run that are recognized with high-confidence as different image classes by a high-performing deep neural network trained on ImageNet [31].”
BackgroundHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 3×
“Policy gradient methods [50], Q-learning [41], genetic algorithms, other variants of ES, diversity-promoting algorithms such as QD [9, 37, 42, 45] and NS [38], including NS-/NSRA-ES [8], and many other such algorithms are all viable alternatives.”
BackgroundMethodReinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 1×
“In reinforcement learning (RL) [59], modest beginnings in pole balancing [1] have given way to learning to play Atari from pixels [3].”
BackgroundRobots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979 · cited 6×
“Novelty search with local competition (NSLC) [37] and MAP-Elites [9, 42] rectify this limitation by adding a quality pressure (hence ushering in QD in earnest), but they still inevitably hit the limit of what is possible to discover in the particular environment where they are run–there are only so many novel ways a h…”
BackgroundMethodInfluentialSelf-citationAbandoning Objectives: Evolution Through the Search for Novelty Alone
Lehman, J., & Stanley, K. O. (2011) · Evolutionary Computation · doi:10.1162/evco_a_00025 · Cited by 917
Self-citationCompositional pattern producing networks: A novel abstraction of development
Stanley, K. O. (2007) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-007-9028-8 · Cited by 713 · cited 1×
“Much more flexible, or even unbounded environment encodings, such as an indirect encoding like compositional patternproducing networks (CPPNs) [56], can potentially enable POET to traverse a far richer problem space.”
BackgroundSelf-citation
Research works printed but not linked: 25
- Bedau Mark, 2008, Proceedings of the Eleventh International Conference on Artificial Life (Alife XI), Seth Bullock, Jason Noble, Richard Watson, and Mark Bedau (Eds.). MIT Press
- Brockman Greg, 2016, arXiv:arXiv:1606.01540 · arXiv
- Florensa Carlos, 2018, International Conference on Machine Learning. 1514--1523
- Florensa Carlos, 2017, Proceedings of the 1st Annual Conference on Robot Learning. 482--495
- Forestier Sébastien, 2017, Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint arXiv:1708.02190 · arXiv
- Gupta Abhishek, 2018, Unsupervised Meta-Learning for Reinforcement Learning. arXiv
- David Ha., 2018, Reinforcement Learning for Improving Agent Design. arXiv preprint arXiv:1810.03779 · arXiv
- Heess Nicolas, 2017, Emergence of Locomotion Behaviours in Rich Environments. CoRR abs/1707.02286 · arXiv
- Justesen Niels, 2018, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi.
- Kingma Diederik, 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 · arXiv
- Langdon W. B., 2005, Pfeiffer - A Distributed Open-ended Evolutionary System. In AISB'05: Proceedings of the Joint Symposium on Socially Inspired Computing (METAS 2005
- Lehman Joel, 2011, Stanley
- Matiisen Tambet, 2017, Teacher-Student Curriculum Learning. arXiv preprint arXiv:1707.00183 · arXiv
- Mnih Volodymyr, 2016, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
- Mouret Jean-Baptiste, 2015, Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 · arXiv
- Popovici Elena, 2012, De Jong
- Pugh Justin K, 2016, Stanley
- Salimans Tim, 2017, Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 · arXiv
- Savinov Nikolay, 2018, EPISODIC CURIOSITY THROUGH REACHABILITY. arXiv preprint arXiv:1810.0227
- Schmidhuber Jürgen, 2013, Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology 4
- Schulman John, 2015, International Conference on Machine Learning. 1889--1897
- Soros L.B., 2016, How the Strictness of the Minimal Criterion Impacts Open-Ended Evolution. In ALIFE 15: The Fifteenth International Conference on the Synthesis and Simulation of Living Systems. 208--215
- Stanley Kenneth O., 2017, Open-endedness: The last grand challenge you've never heard of. O'Reilly Online
- Wiegand R. Paul, Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation (GECCO'01)
- Zhang Xingwen, 2017, Stanley
Cited by
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
“This is a very interesting idea encompassing papers such as POET [11], Generative Teaching Networks [12], and the Synthetic Petri Dish [13] which describe algorithms that learn the environment to learn from.”
BackgroundMethodDeep problems with neural network models of human vision
Bowers, J. S., Malhotra, G., Dujmović, M., Llera Montero, M., Tsvetkov, C., Biscione, V., Puebla, G., Adolfi, F., Hummel, J. E., Heaton, R. F., et al. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002813 · Cited by 204
A biological perspective on evolutionary computation
Miikkulainen, R. P., & Forrest, S. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00278-8 · Cited by 129
“In such a setting, collaborative co-evolution can create cooperative structures that work together 23,69 , and competitive co-evolution can dynamically produce new challenges at different levels 102,114 .”
BackgroundEvolutionary Reinforcement Learning: A Survey
Bai, H., Cheng, R., & Jin, Y. (2023) · Intelligent Computing · doi:10.34133/icomputing.0025 · Cited by 136
“The paired open-ended trailblazer (POET) algorithm has been proposed to generate increasingly complex environments and optimize their solutions concurrently by combining the methods of NS, MAP-Elites, and minimal criterion coevolution [170, 171].”
MethodLanguage Model Crossover: Variation through Few-Shot Prompting
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., & Lehman, J. (2024) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3694791 · Cited by 75
“multi-objective EAs [15, 18], evolutionary strategies [1, 3], or in support of open-ended evolution [85].”
Self-citation
Metadata
- Authors
- Wang, R., Lehman, J., Clune, J., & Stanley, K. O.
- Affiliations
- Uber AI (United States) (US)
- Venue
- Proceedings of the Genetic and Evolutionary Computation Conference
- Publisher
- ACM
- Published
- 2019-07-13
- Pages
- 142-151
- OpenAlex
- W2954700257
- Semantic Scholar
- ac6449fc71c8768e654a5cdd5f3b3fa230fbb523
- Cited by
- 61
- References
- 35
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 12 · 2023: 8 · 2022: 11 · 2021: 9 · 2020: 10 · 2019: 3
- FWCI
- 3.16
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Genetic and Evolutionary Computation Conference (GECCO) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Evolution and Genetic Dynamics
- Keywords
- open-endedness, quality diversity, novelty search, MAP-Elites, environment generation, solution transfer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: POET: open-ended coevolution of environments and their optimized solutions. https://app.cochat.ai/papers/10.1145/3321707.3321799
- The paper
- Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019). POET: open-ended coevolution of environments and their optimized solutions. Proceedings of the Genetic and Evolutionary Computation Conference. https://doi.org/10.1145/3321707.3321799
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.